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Abstract 


In support of the NASA Aviation Safety and Security Program's Synthetic Vision Systems Project, a series of 
piloted simulations were conducted to explore and quantify the relationship between candidate Terrain 
Portrayal Concepts and Guidance/tunnel Symbology Concepts, specific to General Aviation. 

This work was the second part of a three-part study related to the Symbology Development for Head-Down 
Displays test series. The objective of the these experiments were to determine if there were any interactions 
between terrain portrayal and guidance symbology concepts over a spectrum of these, as well as to confirm 
that increases in pilot performance due to guidance symbology concepts were preserved with the addition of 
Synthetic Vision Systems terrain. The focus of this experiment was on an advanced application of this 
technology to fully exploit its’ potential for safe operations during a low altitude en route maneuver in 
Instrument Metrological Conditions in the central mountains of Alaska (Merrill Pass). These types of 
operations are not practical with current technology. A total of 18 general aviation pilots, with three levels of 
pilot experience, evaluated a test matrix of four terrain portrayal concepts and six guidance symbology 
concepts. 

Both quantitative and qualitative measures were recorded and analyzed. Quantitative measures included 
various pilot/aircraft performance data, flight technical errors and flight control inputs. The qualitative 
measures included pilot comments and pilot responses to the structured questionnaires such as perceived 
workload, subjective situation awareness, pilot preferences, and the rare event recognition. 

There were statistically significant effects found from guidance symbology concepts and terrain portrayal 
concepts but no significant interactions between them. The lack of significant interactions between guidance 
symbology concepts and terrain portrayal concepts might help Synthetic Vision Systems display designers to 
select terrain portrayal and guidance symbology concepts independent of each other. Lower flight technical 
errors and increased situation awareness were achieved using Synthetic Vision Systems displays, as compared 
to the baseline Pitch/Roll Flight Director and Blue Sky Brown Ground combination. The results indicate that 
all pilots performed very well, mostly within the 75 ft of vertical and lateral limits indicated by one dot of the 
path based course deviation indicators. With the same Synthetic Vision Systems training provided to all three 
pilot groups, low time pilots (with no Instrument Flight Rules rating) performed as well as pilots with 
instrument flight rules rating during this low altitude en-route scenario in simulated instrument meteorological 
conditions with Synthetic Vision Systems displays. Overall, those guidance symbology concepts that have 
both path based guidance cue and tunnel display performed better than the other guidance concepts. 



Introduction 


Limited visibility has been the single most critical factor affecting both the safety and capacity of worldwide 
aviation operations [1]. A goal of the Synthetic Vision Systems (SVS) Project of the NASA Aviation Safety 
Program is to eliminate poor visibility as a causal factor in aircraft accidents as well as enhance operational 
capabilities through application of SVS technology. SVS displays can enhance pilot’s Situation Awareness 
(SA) [1-8] and have the potential to reduce the occurrence of Controlled Flight into Terrain (CFIT) accidents. 
In addition, SVS displays may reduce or eliminate pilots’ Spatial Disorientation (SD), a primary cause of Low 
Visibility Loss of Control (LVLOC) accidents. 

With the integration of Global Positioning Systems (GPS), advanced solid state Attitude and Heading 
Reference Systems (AHRS), and the imagery derived from terrain, obstacle, and airport databases, into 
Primary Flight Displays (PFD), SVS displays provide pilots with clear, day time view of the outside world, 
regardless of weather conditions and time of day [2]. In addition, through the integration of advanced 
symbology (i.e. highway in the sky, Flight Path Marker, etc.), pilot situation awareness and control 
performance is drastically improved with no effective increase in pilot workload. Recently, there have been 
many studies dealing with various components of SVS and the merit of SVS for General Aviation (GA) as a 
whole [5-6]. However, the proper marriage of SVS terrain and guidance symbology technologies into the PFD 
has a profound implication to the overall integrated performance of these displays, and it is of utmost interest 
to the NASA, FAA, and the aircraft/avionics manufacturers as discussed in a recently published FAA advisory 
circular [12]. 

As an initial investigation, the GA element of the SVS Project conducted a simulation study that focused on 
determining the associated benefits of SVS displays towards reducing LVLOC and CFIT accidents for GA 
pilots [5-6]. The study simulated an inadvertent Instrument Meteorological Condition (IMC) encounter during 
basic air maneuvers Results of those simulations demonstrated the effectiveness of SVS as compared to 
conventional GA round dial instrumentation in reducing pilot errors and thus improving pilot’s ability to 
control the aircraft while in IMC. 

The next series of experiments were the Terrain Portrayal for Head-Down Displays (TP-HDD) simulation, 
conducted in the General Aviation Work Station (GAWS) at the NASA Langley Research Center (LaRC), and 
flight experiments, in the Langley Cessna 206 research aircraft [10-12]. The TP-HDD experiment series was 
conducted to examine the effect of combinations of Digital Elevation Model (DEM) resolution, terrain texture, 
and Field of View (FOV) on pilot’s SA, workload, and FTE. Variations of display type were: 1) conventional 
round dials with a pair of Course Deviation Indicators (CDI); 2) baseline PFD (no terrain) with a single simple 
tunnel; and 3) SVS terrain depicted on the PFD with the same tunnel. SVS Terrain Portrayal Concepts (TPC) 
were created through combinations of various digital elevation models and terrain texturing concepts with 
large variations existing with each. The researchers were able to demonstrate the efficacy of SVS displays for 
a comprehensive spectrum of pilots in both mountainous and flat-maritime environments. Pilots preferred 
higher-resolution digital elevation models. Based on pilot preference evaluations, 3 arc-sec digital elevation 
model (DEM) resolution was considered satisfactory, and the 30 arc-sec digital elevation model, while least 
preferred of the digital elevation models investigated, was still considered a substantial enhancement over 
standard gauges. In addition, the field of view of the display device can also influence the effect of SVS 
terrain displays. Among the three field-of- view values (30°, 60°, and 90°) studied, 60° field-of-view was the 
most useful and values lower than 60° were only possible in calm conditions (during the flight tests in the 
Cessna 206). 

Many studies have compared various Guidance Symbology Concepts (GSC) during development of this 
technology. Most of these studies were based on a simple blue-sky over brown ground terrain representation, 
common to conventional primary flight displays [13-16] and, successfully have led to various 
recommendations and guidelines for the design of advanced cockpit displays [20-21]. Consequently, one area 
of interest to SVS researchers was the integration of guidance symbology with the background terrain portrayal 
to minimize the clutter of the SVS displays. While providing much more information to the pilot than a 
conventional blue-sky over brown-ground primary flight display, the effect of SVS terrain on display clutter 
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was not defined prior to this study reported herein, leading to concerns regarding viable SVS guidance and 
symbology concepts [17-18]. 

The general progression of SVS studies have moved from initial higher-altitude operational concepts into low 
altitude terminal and low en route operations in close proximity to terrain [19-21]. 

Motivated by the safety concerns of GA operations in mountainous terrain, in particular in central mountains 
of Alaska [22], a series of experiments was conducted in the General Aviation Work Station that were referred 
to as the ’Symbology Development for Head Down Displays (SD-HDD)’ test series. The objective of the 
these experiments were to determine if there were any interactions between SVS terrain portrayal or guidance 
symbology concepts over a spectrum of these, as well as to confirm that increases in pilot performance due to 
guidance and symbology concepts were preserved with the addition of SVS terrain. 

The first experiment in this series, applied to terminal area operations, evaluated a curved approach and a 
missed approach scenario at Juneau, Alaska. Discussion of the results of this experiment can be found in [23]. 
The second and the third experiments of the SD-HDD concentrated on low altitude en route maneuvers in the 
central mountains of Alaska (Merrill Pass). The second experiment is the focus of this paper. The preliminary 
results of this experiment were summarized in [24]. It is intended that this paper provide the reader with a 
detailed report of the experiment goals and methodology, experiment set-up and scenario, the conduct and 
administration of the experiment, results of objective and subjective measures, and pilots’ response to 
structured questionnaires and their comments throughout the experiment. 
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Method 


The focus of this experiment was on an advanced application of SVS technology to fully exploit its potential 
for safe operations during a low altitude en route maneuver in instrument metrological conditions in the central 
mountains of Alaska (Merrill Pass), not practical with current technology. This section provides a description 
of the experiment apparatus and methodology. When discussing the experiment apparatus, a detailed 
discussion of the display concepts will be presented and the independent variables will be defined. The 
description of the methodology will include discussions related to the experiment scenarios, selection of the 
Evaluation Pilots (EP), training of pilots, the test protocol, and selection of dependent variables. 


Experimental Apparatus 

General Aviation Work Station 

The experiment was conducted in the General Aviation Work- Station, a fixed based flight simulator equipped 
with two separate 6-inch LCD Head down Displays (HDD) as illustrated in Figure 1. The General Aviation 
Work-Station had been successfully used for previous SVS-GA experiments and it has established itself as a 
low cost but effective simulator that allows rapid replacement of display concepts and experiment scenarios. 
The two 6-inch displays were Commercial-Off-The-Shelf (COTS) VGA (640x480) monitors. The left 6-inch 
display served as the SVS-PFD and the right one as a Navigation Display (ND) with multi-level range 
selection capability, developed at NASA LaRC. Pilot selectable fields of view (horizontal angle of the image 
that is presented on the display) of 30 and 60 degrees were available throughout the scenarios for the PFD and 
the current FOV boundaries were depicted on the ND. An SXGA (1280x1024) overhead projector provided 
the Out-The- Window (OTW) view depiction providing approximately 25 degrees vertical and 33 degrees 
lateral field of view, Figure 2. The OTW imagery used in this experiment was based on the same database as 
the photo realistic terrain portrayal, which will be discussed below under the terrain portrayal concepts. The 
quality and validity of OTW projected displays during ground based simulations have been discussed by many 
researchers and their limitations and remedies, which were studied for low speed and hover flight [25], have 
been taken in consideration here. 

The GAWS was based on a modified Precision Flight Controls PC-based Aviation Training Device (PFC- 
PCATD), model PI- 142 instrument trainer, consisting of dual yoke and rudder pedals with a radio stack 
positioned between the two pilot seats. The evaluation pilots flew the scenarios from the left seat. The right 
seat was occupied by the instructor during the training sessions only. The flight simulator software was 
configured with a Cessna C-172 dynamic flight model from Initiative Computing Electronic Instrument 
Training Environment (ELITE*™) Simulation Solutions Company. 
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Figure 1: General Aviation Workstation (GAWS) Set-up 



Figure 2: General Aviation Workstation (GAWS) Side View Schematics 


Basic Symbology for Navigation Display: 

The ND served as a strategic display depicting the top view of the outside world, Figure 3. Ownship position 
was indicated by a white arrowhead in a track up mode. The preplanned route (flight path) and the desired 
waypoints were displayed in magenta color. The boundary of the FOV, selected for PFD, was depicted in 


5 



green to help associate items in the navigation display with the SVS primary flight display. The terrain 
displayed on ND was simply the 2-D overhead view of the same terrain database as used on SVS primary 
flight display (see terrain portrayal concepts below). In addition to the typical ND speed and distance 
information, the wind direction and speed were also displayed. Magnetic heading, track and course were 
labeled on the compass rose. The range and the scale of the viewing area were displayed on the magnetic track 
line. 



Figure 3: Navigation Display (ND) 


Basic Symbology of the Primary Flight Display: 

Basic symbology for all displays included integrated airspeed and altitude tapes, roll scale, etc., typical of 
modem GA primary flight displays (Figure 4). The pitch ladder label was displayed on the left side of the 
viewing area with increments of 5 and 10 degrees, up and down. The bank angle was shown on the roll scale 
on top of the display with increments of 5 and 10 degrees, right and left. A roll pointer with an integrated 
sideslip indicator shows the instantaneous bank angle and below the sideslip indicator the digital magnetic 
heading was displayed. The airspeed and altitude tapes were on either side of the viewing area with a Vertical 
Speed Indicator (VSI) integrated into the altitude tape. The airspeed tape was color-coded based on Cessna 
172 Reference Airspeeds (V-speeds) and the airspeed bug was pre-set to the assigned airspeed. Part of the 
airspeed tape included an airspeed trend indicator that predicted the airspeed change 10 seconds in future. The 
longitudinal attitude (the pitch attitude) was shown by a waterline marker when used with the pitch reference 
scale while the Flight Path Marker (FPM) provided information about the current flight path. The FPM cluster 
included the sideslip indicator, displayed again on the top of the vertical tail of the FPM symbol, and airspeed 
error (the difference between current indicated airspeed and the assigned airspeed). This airspeed error was 
shown as an increasing or decreasing column on left wing of FPM symbol. An acceleration arrow that moved 
up and down from the left wing of FPM symbol was implemented. This arrow indicated acceleration along the 
flight path and could be adjusted by the power setting to maintain the desired airspeed for the given vertical 
path. Linear vertical and lateral path deviations where displayed using the CDI scale with “dog bone” shaped 
indicators. The selected FOV was digitally displayed in the lower left corner of the PFD display area. 
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Figure 4: Basic Symbology of the Primary Flight Display (PFD) 

Terrain Portrayal Concepts 

The distance between elevation data points (post-spacing) for a given database is referred to as the digital 
elevation model resolution. The highest terrain resolution available (2 arc-sec) was selected for the Merrill 
Pass simulations. The smallest triangular polygon capable of being rendered has sides equal to the digital 
elevation model resolution. 

The SVS terrain database generation process used a precompiled method that allocated polygons in areas with 
large terrain variations. Terrain texturing refers to the method used to color the polygons that comprise the 
SVS terrain database. The three primary SVS texturing concepts tested were Constant-Color Fishnet (CCFN), 
Elevation-Based Generic (EBG), and Photo-Realistic (PR). Cultural features, such as roads and rivers, were 
included as objects in the SVS terrain database. A sample comparison of the four terrain display concepts 
utilized in this experiment is shown in Figures 5 a-d. These figures show the same view and orientation of 
Merrill Pass in Alaska, with the following variations; 

1- A standard generic Blue Sky Brown Ground (BSBG) primary flight display served as the baseline 
terrain portrayal concept (Figure 5a). 

2- The constant-color texturing concept was developed to represent a current industry concept. The 
fishnet had 500 ft squares (Figure 5b). 

3- The elevation based texturing concept consisted of twelve equal-height coloring bands that correspond 
to different absolute terrain elevation levels, similar to the colors employed for Visual Flight Rules (VFR) 
sectional charts. Lower terrain levels were colored with darker colors; higher terrain levels were assigned 
lighter colors. A shade of green was set to the sea level elevation. The lightest color was set to the highest 
terrain within a rectangle of 157x180 Nautical Miles (NM) surrounding Merrill Pass, approximately 9,000 ft 
MSL (Figure 5c). 

4- The Photo Realistic texturing concept was derived from full color ortho-rectified 4 m satellite imagery 
data. The resulting scene was a highly realistic view due to the photographic imagery employed. The PR 
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texturing required special graphics hardware due to the amount of texture memory required to render the 
realistic scene at an acceptable frame rate (i.e. > 30 Hz) (Figure 5d). As mentioned earlier, PR concept was 
also used as the out-the-window display which was projected at the GAWS wall with a field of view of 33 
degrees. 




a) Blue Skv Brown Ground 


b) Constant Color Fishnet 




* ^ 


c) Elevation Based Generic d) Photo Realistic 

Figure 5: Terrain Portrayal Concepts 


Guidance/T unnel Symbology Concepts 

As employed for this experiment, Highway-in-the-Sky (HITS) concepts consisted of two main elements. The 
first element was a Three Dimensional (3-D) pathway of various dimensions and physical presentations and 
the second feature was referred to as the guidance cue. One of the most common depictions of a pathway is a 
tunnel. Common variations of tunnels are the shape, size, color, whether the comers are connected, and 
whether the boxes are connected along the flight path, etc. 

Each pathway-based guidance symbology concept employed for this effort represents guidance symbology 
designs developed and used in commercial products as well as other pathway-based guidance symbology 
concepts guidance and symbology concepts extensively evaluated in previous research efforts and proposed for 
future SVS display applications. The approach for this study was to employ each GSC as developed and not 
refine each concept to preserve specific independent parameters (such as tunnel size, color, etc.). The selection 
of guidance and symbology concepts was intended to provide a spectmm of visual complexity from simple 




Pitch/Roll Flight Director (PRFD) command bars up to connected box tunnels with sliding box guidance cues 
and lateral predictors. 

In addition to the tunnels depicting the flight path, path based course deviation indicators were also 
implemented. The limits of the CDI values were set to +/-150 ft, with 2 dots indication (75 ft per dot) for a full 
deflection in both horizontal and vertical directions from the flight path. The limits of the CDI values were 
selected to match the displacement limits (in feet) at the missed approach point (MAP) during challenging SVS 
approaches investigated in reference 22. In a typical precision instrument approach, below 500 ft Above 
Ground Level (AGL), the aircraft is expected to be established on a stabilized final approach segment with 
very small variations in lateral and vertical deviations from a prescribed flight path. As discussed earlier, the 
experiment scenario was to be a challenging scenario simulating a low en-route flight at 500 ft AGL. The CDI 
scale remained constant for every GSC studied in this experiment, even though tunnel sizes varied. This 
ensured that as long as CDI limits were not exceeded the own ship position remained within the boundaries of 
all four guidance and symbology concepts with a tunnel, as described below. Guidance cues selected were 
based on currently available technology and were 1) a split cue Pitch/Roll Flight Director; 2) a single cue 
Ghost Plane; and 3) a single cue guidance box with laterally quickened Flight Path Marker. There were four 
general categories of Guidance/Tunnel Symbology Concepts, comprising combinations of different guidance 
cues and Tunnels, as shown below: 

A - Guidance Cue, No Tunnel 

1- Pitch/Roll Flight Director 

2- Ghost Plane (GP) 

B- No Guidance Cue, With Tunnel: 

3- Unconnected Boxes Tunnel (UBT) 

4- Crows Feet Tunnel (CFT) 

C- Guidance Cue, With Tunnel: 

5- Crows Feet Tunnel with Ghost Plane (CFTGP) 

6- Connected Boxes Tunnel (CBT) with Guidance Square and laterally quickened Flight Path Marker 
D-No Guidance/No Tunnel: 

7- Comparison of the three SVS terrain portrayal display concepts Constant Color Fishnet, Elevation 
Based Generic, and Photo Realistic (in simulated IMC) with the baseline PFD, Blue Sky Brown Ground, in 
simulated Visual Meteorological Condition (VMC). 

Each one of the above Guidance/Tunnel Symbology Concepts are described in detail below and illustrated in 
Figures 6 through 11. 


Guidance/T unnel Symbology Concept: Pitch/Roll Flight Director 

For the Pitch/Roll Flight Director the displacement of the horizontal and vertical error bars (magenta in color) 
from the water marker indicated the commands for pitch and roll, respectively, to correct for path errors [26]. 
Examples of the block diagrams and control logics for guidance and symbology concepts are shown in 
Appendix A. For the present study the time constants for both roll and pitch directors were set to 0.25 
seconds. 

To achieve precise path control (to center the CDI symbols), evaluation pilots were instructed to position the 
water-marker on the horizontal and vertical bars of PRFD, thus bringing the bars to a cross-hair shape. 
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Figure 6: Pitch/Roll Flight Director 


Guidance/T unnel Symbology Concept: Ghost Plane 


In this concept, a symbolic aircraft was depicted on the PFD moving 5 seconds ahead of own ship on the 
projected tangent of the current flight path [27]. The Ghost Plane shows the path correction necessary to get 
back on the desired flight path. To achieve precise path control (to center the CDI), evaluation pilots were 
instructed to position the Flight Path Marker on the Ghost Plane. 



Figure 7: Ghost Plane 
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In extreme cases where the pilot was far off course, the GP remained pegged at the comer of viewing area, 
changed color to amber, and a line was drawn from Flight Path Marker to the GP indicating the direction of the 
required correction. 

Guidance/T unnel Symbology Concept: Unconnected Boxes Tunnel 


This tunnel concept was implemented during the TP-HDD experiment which was adopted from an industry 
concept that was used in the FAA’s Capstone-2 Project [28]. The comers of the green boxes were connected 
but the boxes were not connected to each other. Each tunnel box was 400 ft wide and 320 ft tall. The boxes 
extend to one nautical mile and the number of the boxes was directly related to the field of view selected. For a 
field of view of 60° there were 7 boxes in a mile. These uniformly placed rectangles depicted the desired path. 
For a curved path the boxes were tilted to 10° when the radius of the turn required a bank angle of more than 
5°. There was no path based guidance cue for this concept. For precise flight path control, the evaluation pilots 
were instmcted to position the Flight Path Marker within as many oncoming boxes as possible to ultimately 
center the CDIs. This concept also included 3-D visualization of waypoints as tethered balloons on the PFD. 



Figure 8: Unconnected Boxes Tunnel 


Guidance/T unnel Symbology Concept: Crows Feet Tunnel 

The Crows Feet Tunnel concept was developed at NASA LaRC in conjunction with the Ghost Plane concept. 
The corners of the magenta boxes were not connected and were shaped in three dimensions, resembling crows’ 
feet. The CFT dimensions were a constant 600 ft wide and 350 ft tall. The evaluation pilots were instmcted to 
place the Flight Path Marker in the middle of the oncoming boxes to center the CDIs. 
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Figure 9: Crows Feet Tunnel 


Guidance/T unnel Symbology Concept: Connected Boxes Tunnel & Guidance Square 

This concept was based on an industry research concept in which the white boxes defining the pathway were 
connected to each other and continued to the final waypoint. The tunnel disappeared after three nautical miles 
distance, also known as the total fade distance. In addition to the 300 ft by 300 ft tunnel, there was a magenta 
guidance square moving 5 seconds ahead of own ship providing the guidance cue [29]. As an integral part of 
this concept, the Flight Path Marker has a quickening of 5 seconds in the lateral direction thus it was referred 
to as a path predictor. For precise flight path control, evaluation pilots were instructed to use the tunnel and 
position the predictor in the center of the guidance square. 


12 




Predictor, similar to V V 
But 5 seconds ahead in 
Lateral direction 


Guidance Box 
5 seconds ahead 


Figure 10: Connected Boxes Tunnel 


Guidance/T unnel Symbology Concept: Crows Feet Tunnel with Ghost Plane 

This guidance and symbology concept was a combination of the CFT and GP concepts and (as described 
above) developed at NASA LaRC. The evaluation pilots were instructed to follow the tunnel and position the 
Flight Path Marker on the GP for precise flight path control. This GSC provides both a path based guidance 
cue and de-emphasized tunnel to manage visual clutter. 



Figure 11: Crows Feet Tunnel with Ghost Plane 
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Guidance/Tunnel Symbology Concept: No-Guidance/No-Tunnel 

This concept incorporated variations of terrain portrayal concepts on the Primary Flight Display, without any 
tunnel or guidance cues (see Figure 5). The combination of the Blue Sky Brown Ground display with no 
guidance and no tunnel during IMC was removed from the test matrix, as this case would not be practical. 
Instead, the baseline for no guidance no tunnel PFD (Blue Sky Brown Grown) was flown in simulated VMC. 
Thus, the three terrain portrayal concept runs with no guidance or tunnel in IMC were compared to BSBG runs 
in VMC. In fact, this combination (BSBG with no guidance and no tunnel used in simulated VMC) served as 
the baseline for when comparing use of any of SVS display combinations in IMC to VMC operations. 


Experiment Scenario 

In Alaska, many populated areas are surrounded by high mountainous terrain and are accessible only by air or 
sea. This has generated a high-reliance on GA aircraft as the main mode of transportation. Frequent low 
altitude icing levels exist in this region, and when combined with no low-level IFR infrastructure, it makes 
traditional Instrument Flight Rules (IFR) operations impossible for most of the GA aircraft in service. Pilots 
often find themselves navigating through treacherous mountain passes, in an attempt to remain in VMC but 
sometimes in very poor visibility, to complete their flights. There have been many accidents in these areas, 
especially around Merrill Pass. Thus, Merrill Pass and central mountains of Alaska were selected as a 
challenging low altitude en route scenario for this experiment in an effort to determine the extent that these 
types of operations could be improved by SVS technology. At Merrill pass, the terrain rises towards the pass, 
requiring the pilot to recognize the upslope from terrain depiction to climb through the pass. After passing 
through the crest of the mountain, the terrain slopes down towards the glaciers. In Alaska, pilots flying 
Alaskan mountain passes traditionally use “rules of the road” staying on the right side of the mountain path. 
These operations are both at higher altitude than simulated in this experiment and they are normally conducted 
in visual metrological conditions. After extensive consultations with numerous Alaskan pilots and air taxi 
operators, the experiment scenario was designed to be a) as challenging as possible, b) simulating an 
inadvertent transition from VMC to IMC, and c) it was assumed that in future aircraft equipped with SVS 
would be also capable of advanced traffic separation such Automatic Dependent Surveillance-Broadcast 
(ADS-B) . Therefore, the flight path was designed to follow the center of the pass. Maintaining path position 
was essential in maintaining adequate terrain clearance. 


Primary Test Scenario 

The scenario started with the aircraft flying at 500 ft AGL and Indicated Airspeed (IAS) of 100 kt in VMC. A 
transition to IMC was simulated at one minute into the scenario. The EPs task was to maintain 500 AGL while 
monitoring the path based course deviation indicators. The planned flight path for experiment scenario was 
designed to follow the terrain at an altitude of 500 ft AGL at all times. This path could provide an optimal 
altitude for search and rescue operations or a suitable altitude to avoid potential icing conditions. As illustrated 
in figure 12 the scenario terminates after passing through Merrill Pass. For better realism and added workload 
(typical of mountain flying), light turbulence and wind (variable both in speed and direction) were also 
simulated. 
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Figure 12: Experiment Scenario, Merrill Pass, AK 

Rare Event Scenario 

The rare event scenario was performed with the NGNT condition. Unknown to the evaluation pilot, one minute 
into the flight, the engine power was linearly reduced to a level (minimum) corresponding to 25% of max 
output without any changes to the simulated engine noise. The intent was to simulate degradation of aircraft 
performance due to structural icing. In all rare event runs, the performance degradation process started one 
minute into the flight and the power level reached its minimum within one minute. The purpose of this rare 
event scenario was to see if the presence of the terrain on PFD affected the evaluation pilot’s situation 
awareness of the imminent emergency conditions. During the pilot briefing and pilot training (see Appendix 
B) evaluation pilots were reminded to a) “fly the simulation just like real flying avoiding any hazardous terrain 
of flight situations when they occur” and b) “communicate their intentions and take corrective action if 
necessary”. 

As it will be discussed later in the “Dependent Variable” section, various new variables particular to the rare 
event scenario were recorded. These included the time at which the evaluation pilot noticed the impending 
problem, 2) the time at which EP initiated any corrective action 3) the outcome of his/her corrective action. 
Based on the actions taken by the evaluation pilots, the outcome data was categorized into the following 
possibilities: 1) was EP successful in avoiding a controlled flight into terrain?, 2) did EP reverse the course and 
make a 180 degree-tum back to lower terrain?, 3) was there a loss of control?, and 4) was there a CFIT as a 
result of 180 degree-tum? Special software was developed to provide the experiment observer an instantaneous 
readout of all aircraft performance data and distance from the terrain. Accordingly, a LOC was defined as 
simultaneous large excursions in pitch (more than 30 degrees), bank (more than 60 degrees) or speed 
(increasing). A CFIT was declared when EP reached zero height, AGL, without any attempt to avoid the 
terrain or land on the terrain in a landing attitude. Of course, verbal acknowledgements of LOC or CFIT by the 
evaluation pilots were also utilized as other criteria. 


Evaluation Pilots 

Three groups of evaluation pilots, a total number of 18, were recmited from around the country representing a 
comprehensive sampling of the GA pilot spectmm. The grouping criteria were based on the earlier studies and 
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results of GA studies on the subject of relation between accident rates and pilot flight experience [30 and 31]. 
These studies indicated that VFR-only pilots with total flight time (FT) less than 400 hours had the highest rate 
of accidents. 

Accordingly, the first group of evaluation pilots (VFR group) consisted of 6 low-time pilots, each with less 
than 400 hours of FT and no formal instrument training beyond that required for the FAA private pilot’s 
license (mean FT=175 hours; mean age=44 years). The second group (IFR) consisted of pilots with less than 
1000 hours of FT and with an Instrument Rating (mean FT= 450 hours; mean age=38 years). The third group 
of evaluation pilots (H-IFR) consisted of professional pilots from NASA, Boeing, FAA, and Alaska 
commercial operators with several thousands of hours FT each (mean FT= 8574 hours; mean age=56 years), 
see table 1 for details. 


Pilot Mean Age by Pilot Mean Hours by 

Category Age category Flight Hours Pilot category 


HIFR 

57 


3000 


HIFR 

53 


1560 


HIFR 

62 

56 

14000 

8574 

HIFR 

50 

7482 

HIFR 

56 


17000 


HIFR 

55 


8400 


IFR 

43 


600 


IFR 

38 


850 


IFR 

38 

38 

200 

450 

IFR 

54 

500 

IFR 

24 


200 


IFR 

29 


350 


VFR 

25 


86 


VFR 

40 


100 


VFR 

52 

44 

250 

175 

VFR 

57 

183 

VFR 

43 


64 


VFR 

48 


366 



46 


3066 


Table 1: Evaluation Pilots’ Qualification and Aeronautical Experience 


Training of Evaluation Pilots 

Before the start of the experiment, each pilot received an experiment briefing followed by a pilot briefing 
similar to a Ground School, as well as one-hour of simulator training in the GAWS with an FAA certificated 
flight instructor for instruments. The purpose the briefing and the training was to familiarize each EP with the 
objectives of the experiment and educate the subjects on the salient features of the symbology and simulator 
functionality. An FAA-style training syllabus was utilized for training and for testing of EP’s skills according 
to the FAA Practical Test Standard (PTS) for Private Pilot License. Additionally, evaluation pilots were 
instructed to 1) Use all display information to minimize pilot flight technical errors; 2) Avoid hazardous terrain 
or flight situations; and 3) Communicate their intentions and take corrective action when encountering 
hazardous situations. A sample lesson of the training syllabus is included in Appendix B. 


Test Protocol 


Each EP participated in two 8-hour days of testing, conducting 56 runs of 5-minute duration and one rare event 
scenario. To avoid pilot fatigue all runs (including repeat runs) were randomized and short breaks of 10 
minutes (every hour) and longer breaks of 20 minutes duration (every two hours) were scheduled. In addition, 
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a lunch break of at least one hour duration per day was allocated for every EP. At the end of each run, the 
subject pilot was required to complete a set of subjective run questionnaires administered on a Tablet Personal 
Computer. The between-run questionnaires solicited the evaluation pilot’s subjective estimates of situation 
awareness and perceived workload for each one of the 56 (4 TPC x 7 GSC x 2 Replicates = 56) displays 
immediately after exposure to each display. The general schedule was: Day 1; Pilot briefing, overall training, 
data collection for all guidance/tunnel concepts, Day 2; Data collection continued in the morning. In the 
afternoon, data collection continued for NGNT runs and finished with the rare event scenario. During the exit 
interview several sets of questionnaires related to EP preferences, workload, and SA were administered. All 
formal testing was audio and video recorded for future correlation to the actual data. 


Dependent Variables 

The dependent variables were both objective and subjective measures. 


Subjective Measures: 

The subjective data measures included a set of three batteries of between-run questionnaires, a battery of block 
questionnaires for each block of the experiment, and a battery of exit questionnaires administered at the 
conclusion of the experiment. 

Between Run Questionnaires: 

The first set of between-run questionnaires dealt with perceived situation awareness (see Figure 13.). To have 
an uninterrupted and complete FTE data set, it was decided to use a post-run Situation Awareness Rating Scale 
Technique (SART), [32]. Accordingly, a 3-D SART was computed from Figure 13 and is reported as the 
calculated Situation Awareness according to the following formula: 

SA Calculated = Understanding - (Demand-Supply) 

The set of questions related to perceived workload are shown in Figure 14. For this report, a standard 6-degree 
NASA- workload index (TLX Score) was computed as: 

Score = (Mental +Physical + Temporal + Performance + Effort + Frustration) / 6. 
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Figure 13: Scoring of Situation Awareness Rating (SART) 
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Figure 14: NASA Task Load Questionnaire 


Block Questionnaires: 

In addition to the above between-run-questionnaires, at the conclusion of all runs there were two more blocks 
of questionnaires administered. These questionnaires dealt with overall evaluation pilot situation awareness 
and workload when using the displays with guidance and tunnel (block one) and when using NGNT (Block 
two). At the conclusion of the two block questionnaires another set of questions were posed which dealt with 
EP preferences for any of the display combination used in this experiment. The copies of block questionnaires 
and preferences questionnaires that were administered at the end of EP visit are shown in Appendix D. 


Objective Measures: 

Even though numerous types of objective measures such as pilot control inputs, pilot/aircraft performance 
data, etc., were recorded, this report focuses on the vertical and lateral path errors, which are also referred to as 
Flight Technical Error (FTE). The FTE values were computed by using two different treatments. In the first 
treatment, the entire scenario was treated as a single segment. Note that each run scenario consisted of three 
distinct segments: the first segment was a straight and level flight; the second segment consisted of a shallow 
bank with a steep climb, and the third segment consisted of a moderately banked turn completed with a rollout. 
In the second treatment, the objective measures (i.e. FTE) were computed for all three segments separately. 

The purpose of the segmented treatment was to see if any significant interactions between TPC and GSC could 
be obtained during each one of those distinct maneuvers (segments) with different overall levels of workload 
induced by variations in the amount of required maneuvering. 


19 



Objective Measures for the Entire Run: 

To obtain the EP flight technical errors for an entire run, time history data of all runs were analyzed and 
pertinent statistical parameters such as minimum, maximum, Root Mean Square (RMS), Standard Deviation 
(Std), etc. were computed for all objective measures including vertical and lateral flight path errors. However, 
this report focuses on RMS of vertical and lateral flight path errors. Computations were performed for a 
constant distance of 6.5 NM (1.80-8.3) which corresponds to a Waypoint (WP) near the start of IMC phase of 
flight (WP1) to a waypoint near the end of scenario, after clearing the Pass (WP8). Refer to Appendix C for 
more information about the waypoint locations. Both objective data and subjective measures (gathered from 
each run) were statistically analyzed over the participating EP population using statistics SPSS software 
package SPSS™ by SPSS, Inc. Multivariate Analysis of Variance (ANOVA) and Post Hoc analyses of all 
measures were performed. 

Objective Measures for the Segmented Treatment: 

In the second treatment, the objective measures (i.e. FTE) were computed for all three segments separately. 
Details of the segmentation methodology are also discussed in Appendix C. Computations were performed for 
three segments with following attributes. The first segment had a length of 3.08 NM which corresponds to a 
route from a waypoint near the start of IMC phase of flight (WP 1=8.3) to a waypoint at the start of the first 
climbing turn (WP3=5.22). The second segment had a length of 2.64 NM from the waypoint at the start of the 
first climbing turn (WP3) to the waypoint before the start of the level turn into the mountain gap (WP6=2.578). 
The third segment had a length of 0.778 NM, which started at the waypoint for the level turn before the 
mountain gap (WP6) and into a distance 1.8 near the end of scenario, after clearing the Pass (WP7=1.882). 

The starting point for segment 1 and the end point for the segment 3 were the same points as those selected for 
the start and end of the complete scenario (un-segmented treatment), respectively. 

Additional Objective Measures for the Rare Event: 

For the rare event scenario, in addition to the above objective and subjective measures (after each run), the 
following data were also recorded: 1) The time at which the evaluation pilot noticed the impending problem, 

2) the time at which EP initiated any corrective action 3) the outcome of his/her corrective action. Based on 
the actions taken by the evaluation pilots, the outcome data was categorized into the following possibilities: 1) 
was EP successful in avoiding a controlled flight into terrain?, 2) did EP reverse the course and make a 180 
degree-tum back to lower terrain?, 3) was there a loss of control?, and 4) was there a CFIT as a result of 180 
degree-tum? Special software was developed to provide the experiment observer an instantaneous readout of 
all aircraft performance data and distance from the terrain. Accordingly, a LOC was defined as simultaneous 
large excursions in pitch (more than 30 degrees), bank (more than 60 degrees) or speed (increasing). A CFIT 
was declared when evaluation pilot reached zero height, AGL, without any attempt to avoid the terrain or land 
on the terrain in a landing attitude. Of course, verbal acknowledgements of LOC or CFIT by the evaluation 
pilots were also utilized as other criteria. As it will be discussed in the results section of this paper, there were 
no LOC incidents observed during the rare event scenario. 
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Results 


The results of the study are presented in this section and are grouped in the following manner. First the 
objective and subjective measures that were gathered during and after each run will be statistically summarized 
and discussed in detail. Next the results of the structured block questionnaires will be presented, followed by 
results of the evaluation pilot preferences questionnaires and the summary of EP comments. The focus of the 
results of the objective measures will be on the results of flight technical errors for the each run either as a 
complete segment or as a segmented treatment when the run is partitioned into three arbitrary segments. Prior 
to the presentation of the results of objective and subjective measures for the complete scenario, the 
methodology for handling the flight technical errors is discussed first. 


Runs Excluded from Flight Technical Errors 

There were no “bad” data in this experiment due to technical problems. All runs were included in the statistical 
analyses of the subjective measures. Only during the computation of the flight technical errors, these unusual 
runs were treated as the statistical outliers, as some of these runs either were not completed or evaluation pilots 
did not follow the prescribed flight path during the NGNT runs. These FTE values were removed and replaced 
by the values of the replicate runs. None of the values of the subjective measures were treated as outliers. 

Among the 864 runs (6 guidance symbology concepts * 4 terrain portrayal concepts * 2 replicates * 18 
evaluation pilots) conducted for the cases with guidance and tunnel, there were 4 actual CFIT incidents and 8 
runs where the evaluation pilots encountered temporary loss of situation awareness without LOC or CFIT. 
Table E.l in Appendix E shows the details of these unusual runs, including comments recorded by the 
experiment conductor. The table indicates that, among terrain portrayal concepts tested here, there were no 
unusual runs for the EBG. However, there were 6 unusual runs for BSBG, 4 for CCFN, and 3 for PR. Among 
the guidance symbology concepts, one can see that evaluation pilots had problems with PRFD (5 unusual 
runs), 5 for CFT, 3 for GP, and none for UBT, CBT or CFTGP. As it will be shown in the next sections UBT, 
CBT and CFTGP were also the most favored guidance symbology concepts by the evaluation pilots. 
Consistently, EBG was the most favored terrain portrayal concept by evaluation pilots. 

As mentioned earlier, the subjective measures were designed to be non-intrusive to pilot performance and, by 
design, the questionnaires were administered after the completion of each run and could only be applied to the 
entire scenario. Therefore, the remaining question was if there were any interactions between terrain portrayal 
concepts and guidance symbology concepts for any of the three segments of the scenario base on the resulting 
flight technical errors as the entire scenario was compromised of various levels of required maneuvering. For 
the segmented treatment of FTE, all pertinent measures in each segment were reexamined and the outliers were 
removed from the corresponding segments. Appendix E shows the frequency plots of the segmented measures 
before and after the removal of the outliers. As discussed in the appendix, the second tier outliers were based 
on a single run. In this particular run, the evaluation pilot mentioned that he lost his lateral situation awareness 
as he was “distracted by focusing too much on the navigation display.” Details of these discussions and other 
evaluation pilot comments can be found in the section under Highlights of EP Comments and Appendix F. 

For no guidance no tunnel runs (4 terrain portrayal concepts * 2 replicates * 18 evaluation pilots), Table E.2 in 
Appendix E shows the list of the unusual runs among these 144 NGNT runs. There were 4 CFIT runs with 
CCFN and 2 with EBG. Among the unusual NGNT runs, there were not any cases with PR (in IMC) or BSBG 
in VMC. It means that evaluation pilots using Photo Realistic terrain display concept in simulated IMC were 
able to avoid any CFIT just like those who used the baseline primary flight display in simulated VMC. 
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Results of Objective and Subjective Measures for Complete Scenario 


The results discussed here will include both objective and subjective measures. The focus on the RMS of the 
FTE will be on lateral and vertical path errors and focus of subjective measures will be the results of SART 
and TLX averaged over the particular evaluation pilot population. First plots of the variations of both 
subjective and objective independent variables (as a function of independent variables terrain portrayal 
concepts, guidance symbology concepts, and pilot ratings) will be presented and discussed. In the second level 
of analyses, their interactions between the variables will be presented and discussed. Except view special 
cases, only the results of the statistically significant results will be discussed in the following. 


Effect of Terrain Portrayal Concept 

The particular type of terrain displayed on the PFD affected both pilot objective and subjective measures. 
Analyses of variances were performed on these measures and all post hoc groupings discussed in this report 
were based on the Student, Newman, Keuls method. The mean RMS of the Lateral Path Deviation (LPD) 
showed a statistically significant effect due to levels of terrain portrayal concept, (F(3,720) =5.009, P<.05), 
creating two post hoc subgroups. In the first group, the two concepts with the smallest LPDs were Elevation 
Based Generic and Photo Realistic concepts. The Constant Color Fishnet and Blue Sky Brown Ground 
concepts belonged in the other subgroup. Figure 15 illustrates the bar charts of the mean RMS of LPD. It can 
be seen that the LPD values were approximately 8 ft less for PR and EBG than CCFN and BSBG. This 
indicates that using higher fidelity terrain such as PR or EBG not only improves pilot situation awareness (see 
SART results below) but also can invoke lower values for lateral flight path deviation than when using CCFN 
or baseline BSBG. 



TPC 


Figure 15: Mean RMS of Lateral Path Deviation 

The RMS of Vertical Path Deviation (VPD) showed a statistically significant effect of TPC, (F(3,720) = 4.24, 
P<.05), creating two subgroups (Figure 16). Here, vertical path deviation values were statistically lower for 
EBG than the other 3 terrain portrayal concepts (PR, CCFN, and BSBG) with a mean of 25.5 for EBG versus 
28.5, 30.1, and 30.2 for PR, CCFN, and BSBG, respectively. Again, while the difference in VPD between 
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EBG and the other terrain portrayal concepts may not be operationally significant, it does indicate a pilot 
response to the terrain stimulus with reductions in VPD associated with an increased amount of SA and 
decreased workload (see subjective results below). The EBG concept provides terrain in a somewhat coded 
yet highly intuitive manner providing the pilot with useful information, (e.g., where is the high terrain?), and 
eliminating secondary information, (e.g., whether the terrain is rock or soil). 



TPC 


Figure 16: Mean RMS of Vertical Path Deviation 


Effect of terrain portrayal concepts on situation awareness is shown by the SART data in figure 17. The 
statistically significant results, (F(3,720) = 31.5, p<.05), reveal that there were 3 subgroups formed. BSBG 
had the lowest SA; CCFN was in the middle; and EBG and PR providing the highest levels of SA. The value 
of perceived situation awareness for EBG and PR was almost twice the value for the other two terrain portrayal 
concepts. 

The results of subjective workload, as indicated by TLX data, are provided in figure 18. The effect of terrain 
portrayal concept on TLX data was statistically significant, (F(3,720) = 19.5, p<.05), with two subgroups 
emerging from the post-hoc analyses. EBG and PR provided the lowest perceived workload with CCFN and 
BSBG being in the other group. This is an important result as it indicates that situation awareness was 
increased while workload was actually decreased through the integration of SVS terrain. Interestingly, as it 
will be shown later in section for NGNT data, both SA and TLX values are almost equal or better than to those 
obtained for baseline BSBG in simulated VMC. 
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Figure 17: Perceived Situation Awareness, SART 
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Figure 18: Perceived Workload, TLX 


Effect of Guidance Symbology Concept 

The overall effect of guidance symbology concepts on objective and subjective measures are discussed here. 
These concepts included Pitch/Roll Flight Director, Unconnected Boxes Tunnel, Crows Feet Tunnel with 
Ghost Plane, Ghost Plane with no tunnel, Crows Feet Tunnel without any path based guidance, and Connected 
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Boxes Tunnel. The results of the lateral path deviation for GSC are presented in figure 19. ANOVA results 
for these data (F(5,720) = 108.12, p<.05) indicate that the CBT, CFTGP, and GP concepts were in the first 
subset, UBT in the second, and CFT and PRFD providing the worst performance. It should be noted that the 
best performing guidance and symbology concepts (CBT, CFTGP, and GP) provide lateral path deviations that 
were approximately one-half of the worst group (PRFD and CFT). The magnitude of this result was 
operationally significant and may greatly facilitate accepting new types of operations requiring low flight 
technical errors. 

Mean RMS values of vertical path deviations for guidance symbology concepts are provided in figure 20. The 
effect of GSC on VPD was statistically significant, (F(5,720) = 105.48, p<.05), with post-hoc analysis 
generating five subgroups. The CFTGP and GP generated the lowest vertical path deviation, with CBT being 
the next lowest followed by UBT, PRFD and finally CFT, in that order. Similar to results obtained for lateral 
path deviation, VPD results for the best group (CFTGP and GP) were less than one-half of the worst subset 
(CFT). 

The promise of a tunnel/Flight Path Marker combination presentation is that the pilot, in a glance, can observe 
own ship position, orientation, and trajectory trend relative to the current and future path. This is in contrast to 
a flight director where awareness has to be built-up and inferred through observation of the CDI, flight director 
symbology, attitude indicator, and probably heading and turn-rate over a period of time. This additional 
information bundle increases pilot SA (see Figure 21) leading to lower workload (Figure 22) and better FTE 
(see above figures 19 and 20). 

The exception among the tunnel concepts seemed to be Crows Feet Tunnel display concept. It is speculated 
that the CFT was perceived by the pilots as a tunnel with fragmented comer elements (crows’ feet) and was 
lacking a focused information bundle, leading to the poorly perceived situation awareness. It should be noted 
that the addition of Ghost Plane guidance cue to the CFT concept (crows feet tunnel with Ghost Plane concept) 
may have moved the focus of the pilots from the comers to the central guidance cue. Unlike the Pitch/Roll 
Flight Director concept, the information was readily available but not perceived as easily or quickly as other 
tunnel concepts. Note that the CFTGP (not CFT) concept was based on a variation of NASA concept 
discussed in [33]. 



GSC 

Figure 19: Mean RMS of Lateral Path Deviation 
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Figure 20: Mean RMS of Vertical Path Deviation 

Results of perceived situation awareness for each guidance symbology concept are presented in figure 21. 
ANOVA results, (F(5,720) = 68.84, p<.05), indicate that the CFT and PRFD belonged to the first subset with 
the lowest perceived situation awareness which is consistent with the above FTE results. However, GP 
belongs to the 2 nd subset, CFTGP to the 3 rd , UBT to the 4 th , and CBT to the last subset, with the values of the 
last subset almost six times as large as the first one, -15 versus -87, respectively. It can be seen from figure 21 
that in general guidance symbology concepts with tunnel provided increased SA. As discussed above, the 
primary exception to that trend was the Crows Feet Tunnel. Path preview, as provided by the more visually 
complex tunnels, was a main reason cited for the SA results. EP comments indicated that the comers of CFT 
were sometimes perceived as multiple cues as some of evaluation pilots had difficulty forming the 3-D image 
of the pathway. The Crows Feet Tunnel required pilot’s increased attention, lowering GSC situation 
awareness. Even though the vertical and horizontal path deviations for CFTGP concept were equal or less than 
those for CBT, perceived SA for CBT shows a higher value. This is consistent with EP comments and results 
of preferences questionnaires (to be discussed in the next sections). The reason for this might be that: a) unlike 
the CFT and CFTGP, the CBT’s white tunnel was displayed as a continuous tunnel (not fragmented path 
information) extending to the destination and b) the size of the CBT’s tunnel was exactly the same as the limits 
of the CDI values and not spread farther out of center of the viewing area. 
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GSC 

Figure 21: Mean of Situation Awareness, SART 



GSC 


Figure 22: Mean Workload Measure, TLX 

The effect of guidance symbology concept on TLX results was statistically significant, (F(5,720) = 37.22, 
p<.05). Five subgroups were formed from post-hoc analysis. The TLX data indicate that CBT and UBT 
produced the lowest perceived workload, CFTGP and GP produced the next higher workload, and the CFT and 
PRFD created the highest workload (Figure 22) as a subgroup. This is somewhat consistent with the results of 
calculated situation awareness as both PRFD and CFT provided the lowest SA, and UBT and CBT provided 
the highest SA. As noted above, pilot comments indicated that the comers of CFT were sometimes perceived 
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as multiple cues with Evaluation pilots having difficulty forming the 3-D image of the pathway. Again, the 
fragmentation of the information, by the crows feet comers about the current and future path, required pilot’s 
increased attention (low GSC situation awareness) and also increased their scanning effort (high GSC 
workload). The contribution of the Ghost Plane in CFTGP, as a path based guidance cue, elevated the 
standing of CFTGP to the second subset with GP, for both SA and TLX. Here again, similar to the SA results, 
CBT showed better results (lower TLX) than CFTGP, due to the issues discussed above. 


Effect of Evaluation Pilot Qualifications 

An ANOVA performed on objective and subjective measures for evaluation pilot qualification levels (Rating) 
showed statistically significant results for LPD, (F(2,720) = 4.64, p<.05); VPD, (F(2,720) = 3.07, p<.05); and 
SA, (F(2,720) = 6.63, p<.05) but not for TLX, (F,2,720)=2.78, p=0.063). 



Rating 


Figure 23: Mean RMS of Lateral Path Deviation 
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Figure 24: Mean RMS of Vertical Path Deviation 



Rating 

Figure 25: Mean of Situation Awareness, SART 

The above results appear to be unusual as the H-IFR pilots with higher perceived situation awareness than 
other pilots were not able to have the same level of flight technical errors. However, as it will be shown in the 
analyses of the variables interactions and the results of the block questionnaires, it is not surprising that H-IFR 
pilots gained increased situation awareness from the advanced displays but were not able to assimilate and 
implement the new concepts as well as the other pilots. As it will be shown below, for example, these pilots 
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were able to have lower flight technical errors when using a familiar guidance concept such as a flight director 
(PRFD) but not when using the new Crows Feet Tunnel. 


Independent Variables Interactions 

Careful study of all interactions showed that there were no statistically significant interactions between TPC 
and guidance and symbology concepts for the independent variables considered above. 

There were no statistically significant interactions between evaluation pilot levels (Rating) and TPC, either. 
However, statistically significant interactions were found between Rating and GSC for objective measures 
LPD (F(l 0,720) = 2.034, p<.05); VPD (F(10,720) = 3.37, p<.05); and also SA (F(10,720) = 1.8, p<.05) but not 
for TLX. 
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Figure 26: Mean of Lateral Path Deviation 
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Figure 27: Mean of Vertical Path Deviation 

As shown above (Figures 26 - 27), H-IFR pilots performed somewhat better using flight director (traditional 
guidance) and only slightly worse than the other two pilot groups for other guidance symbology concepts 
except for CFT. This could be attributed to the effects of the fragmented path information provided by CFT, 
which was the most challenging GSC even for the other two pilot groups. In addition, information provided by 
the visual integration of SVS terrain and advanced guidance symbology seemed to be easier to assimilate by 
the lower-time pilots. This is consistent with the results of the perceived SA (see Figure 25). The H-IFR pilots 
rated their SA higher than other pilots for all concepts except for CFT. It seems that H-IFR pilots appreciated 
(perceived positively) any type of modem guidance compared to baseline display (BSBG with flight director) 
(see also block questionnaires), however they were not able to utilize this increased SA to reduce (improve) 
their flight technical errors at the same rate. Overall, consistent with other two EP groups, H-IFR pilots also 
perceived higher SA with CBT and UBT. 
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Figure 28: Mean of Situation Awareness 
Results of Objective Measures for Segmented Treatment 

As mentioned above, there were no significant interactions between the terrain portrayal concepts and the 
guidance symbology concepts for the enter scenario. The question remained if one would observe any 
interactions between TPC and GSC for a specific segment of the scenario as the entire scenario was 
compromised of various levels of required maneuvering. Thus, the entire run was subdivided into three distinct 
segments and the time history of objective measures was analyzed for each segment. As mentioned earlier, the 
subjective measures were designed to be non-intrusive to pilot performance and, by design, the questionnaires 
were administered after the completion of each run and can only be applied to the entire scenario. Therefore, 
the remaining question is if there were any interactions between terrain portrayal concepts and guidance 
symbology concepts for any of the three segments of the scenario base on the resulting flight technical errors. 
The complete results of FTE for the segmented treatment is shown in Appendix G and only the results of the 
FTE for segment 3 will be discussed here. Segment 3 contained the most challenging maneuver of the 
scenario while the aircraft was maneuvered within Merrill Pass at very close proximity of the surrounding 
terrain. The results of FTE for segment 3 are discussed below (see Figures 29-32). Note that the scale of the 
FTE is expanded to 220 ft to account for the larger RMS of deviations in segment 3. 


Effect of Terrain Portrayal Concepts on Flight Technical Errors for Segment 3 

The effect of TPC on the results of mean RMS of lateral path and vertical path deviations were similar to the 
results obtained for the analyses of the complete scenario (recall Figures 15-16) but they were not statistically 
significant as the flight technical error were dominated by the guidance symbology effect in this more complex 
segment of the flight (see also next section). The intensity of maneuvering tasks during this segment 
introduced a larger degree of variability into the data that overwhelmed the relatively small effects from terrain 
portrayal concept. 


32 


Effect of Guidance Symbology Concepts on Flight Technical Errors for Segment 3 

The effects of the GSC on the segment 3 values of mean RMS of the lateral path deviation (s31aerms) and 
vertical path deviation (s3verms) are plotted in figures 29 and 30, respectively. The results of lateral path 
deviation for GSC effects was statistically significant, F(5, 717)= 90.0 (p<.05). Very similar to the results of 
the complete scenario (Figure 15), Pitch/Roll Flight Director and Crows Feet Tunnel exhibited the highest 
mean RMS of lateral path deviation for this segment. PRFD and CFT placed in the first and second post hoc 
group, respectively, while CBT, GP, and CFTGP were in a third group showing the lowest levels of lateral 
path deviation. Again, the guidance symbology concepts with path based guidance cues outperformed the 
other three guidance and symbology concepts tested here. 



GSC 


Figure 29: Mean of Lateral Path Deviations for Segment 3, s31aerms 

The mean RMS of the vertical path deviations for segment 3 (s3verms) for guidance symbology concepts are 
plotted in figure 26. Here again, the results were statistically significant, (F(5,717)=63.9, p<.05), and the 
overall trend for this segment was the same as the results of the analyses for the complete scenario (Figure 16). 
While the post hoc groupings were somewhat different, overall the CFT shows the highest mean RMS of 
vertical path deviation followed by UBT, PRFD and CBT as a second group. Here again, the two concepts 
with the Ghost Plane as path based guidance cue (CFTGP and GP) resulted in the lowest vertical path 
deviation. This indicates that during complex maneuvers near the terrain (segment 3) the displays with path 
based guidance cue (GP and CFTGP) helped evaluation pilots maintain their vertical track better than the other 
Guidance and symbology concepts. Note that this seems to be in contrast to the SA and TLX results as the 
CBT provided better subjective ratings, however the SA and TLX measures were gathered for the entire 
scenario and not for the segment 3. 
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Figure 30: Mean of Vertical Path Deviations for Segment 3, s3verms 


Effect of Evaluation Pilot Qualifications on Flight Technical Errors for Segment 3 

The mean RMS of the LPDs and VPDs for segment 3 were analyzed compared against the three evaluation 
pilot subgroups and there were not any statistically significant results across EP Ratings. 


Interaction Effects for Segment 3 

The only statistically significant interaction results for segment 3 were obtained for the interaction effects of 
the guidance symbology concepts and Rating. Not only were significant FTE interactions obtained for Rating 
and GSC but also the level of the lateral and vertical path errors were much higher for the various Ratings. 
Here again, similar to the results of the complete scenario (Figures 19 and 20), pilots performed best when 
using guidance and symbology concepts with guidance cue systems (CFTGP, GP and CBT) and weaker, 
resulting in highest FTE when using CFT and PRFD. Within each GSC, the high time pilots performed better 
(less lateral and vertical path deviations) than other evaluation pilots when using flight director, as they were 
more familiar with this type of guidance than less experienced evaluation pilots. However, the same group, H- 
IFR, did not perform as well as the other evaluation pilots when using CFT, consistent with their perceived SA 
rating that was shown in Figure 21. 
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Figure 31: Mean of Lateral Path Deviations for Segment 3 
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Figure 32: Mean of Vertical Path Deviations for Segment 3 
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Results of No-Guidance/No-Tunnel Runs 


As it was shown in both the results of the complete scenario and the segmented analysis of flight technical 
errors, the effect of guidance on pilot performance measures was very strong. The purpose of the No 
Guidance/No Tunnel (NGNT) runs was to remove the influence of the GSC and study the main terrain effects 
on the basic primary flight display. The experiment scenario was the same as previous with-GSC runs. The 
Blue Sky Brown Ground display with the out the window visual display simulating visual meteorological 
conditions was used as a baseline VFR test condition. Here again the evaluation pilot’s task was to fly at 500 
ft AGL but without any path guidance, using terrain information provided by the PFD, ND and OTW display. 

Effect of Independent Variables Subjective Measures 

As it is shown in Figure 33, the calculated situation awareness for terrain portrayal concepts was statistically 
significant, (F= (3,119) =13.0, p<0.05). In fact, the post hoc analysis showed that use of BSBG in VMC and 
PR display in IMC provided the highest SA group while EBG belonged to the second highest SA group. The 
lowest SA group computed was for CCFN, with a negative value. Recall that calculated SA was computed 
from SART = Understanding - (Demand- Supply). Thus, negative value for three dimensional SART means 
that the magnitude of ‘Demand on Attentional Resources’ had been higher than the sum of both 
‘Understanding of the Situation’ and ‘Supply of Attentional Resources.’ 



Figure 33: Mean of SA for Basic SVS (NGNT) 

There were no statistically significant SA results for different evaluation pilot qualifications levels (Rating). 

The perceived workload, shown in Figure 34, was statistically significant, (F=(3,l 19)=7.8, p<0.05) and 
confirmed the same results as those obtained when displays with guidance were used. Here, of course, BSBG 
in VMC (48.29) showed the lowest workload sharing the same post hoc group as PR (54.1) in IMC. Photo 
Realistic terrain portrayal concept also shared the second group with EBG (62.67) while CCFN placed in the 
third group (66.2) and shared this group with EBG. 
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Figure 34: Mean of NASA TLX Score for Basic SVS (NGNT) 

There were no statistically significant TLX results for different evaluation pilot qualification levels (Rating). 


Independent Variables Interactions 

There were no statistically significant interactions between the TPC and EP Rating for subjective measures. 
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Results of Rare Event Scenario 


As discussed earlier, the rare event scenario was a subset of the no guidance no tunnel display combinations. 
Each evaluation pilot was exposed to the rare event scenario only once, at the conclusion of the two-day 
experiment. The order of the terrain portrayal concept for each pilot was randomly selected and care was taken 
to expose the pilot at least once to the particular display (used in his/her rare event scenario) prior to the rare 
event use. As discussed earlier, the purpose of this rare event scenario was to see if the presence of the terrain 
on PFD affected the evaluation pilot’s situation awareness of the imminent emergency conditions. Only two of 
several rare event objective measures are discussed here. The two new measures are the Recognition Time in 
Seconds (TRECS) and the Time to the End of the flight in Seconds (TENDS). 

Figure 35 shows the distribution of TRECS of each rare event run for each evaluation pilot tested. This time 
was measured from the onset of the run to the moment that the evaluation pilot recognized the impending 
emergency. Plotted is also the outcome of the scenario. Those runs that ended with a controlled flight into 
terrain from continuation of flight path were designated as Straight-CFIT (S-CFIT); those CFIT runs that 
occurred after an attempt was made to return to a lower terrain were labeled as U- turn CFIT (U-CFIT); and 
cases where the evaluation pilots performed a successful (intentional) Off-field landing were labeled as 
Straight landing (S-land). As mentioned earlier, evaluation pilots were instructed to communicate their 
intentions when avoiding any hazardous terrain. In Figure 35, the green circle with the label ‘Y’ for COMM 
refers to those pilots who communicated their situation or their intensions to the test conductor. The blue 
circle with the label ‘N’ means these pilots did not communicate. 

All rare event runs were terminated prematurely. Out of the 15 evaluation pilots tested, 12 pilots encountered a 
CFIT straight ahead and two pilots experienced a CFIT during an attempt to return to lower grounds while 
executing a 1 80 degree turn. One pilot recognized the problem and attempted to land safely straight ahead 
utilizing the one mile out the window visibility. Only six evaluation pilots had recognized the impending 
emergency early enough and had time to communicate their intentions before/while taking an action. Note that 
the onset of ‘icing’ simulation occurred after one minute into the experiment run. The full ‘icing’ was 
encountered after additional 60 seconds, i.e. 2 minutes into the flight (see 120 seconds lines in Figure 35). 
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Figure 35: Results of Recognition Time for each EP by OUTCOME and Verbal Communication 
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Results of Rare Event Measures by Terrain Portrayal Concept 

It should be noted that the following results were not statistically significant (F(3,8)=3.2, p<0.09). However, 
the data suggests interesting trends for various terrain portrayal concepts. Figure 36 shows the mean values of 
the recognition time and scenario end time, for different terrain portrayal concepts. Recall that the baseline 
Blue Sky Brown Ground display concept was used with simulated VMC and the results are accordingly 
labeled as BSBGV while the other displays were utilized in a simulated IMC. Therefore, one could consider 
the results of the BSBGV as the time that the pilots were utilizing the out of window visual cues as compared 
to the results of CCFN, EBG and PR which were in simulated IMC where the pilots had to heavily rely on the 
head down displays. Accordingly, it could be concluded that using CCFN without any guidance or tunnel in 
simulated IMC during an emergency, did not help pilots to perform as well as they could have performed if 
they had used baseline display in VMC or if they had used EBG of PR in simulated IMC. In fact the pilots 
who used EBG in IMC were able to recognize the problem (TREC) about the same time as those who used the 
baseline display. Furthermore, these pilots were also able to prolong their flights (TENDS) as long as those 
who used BSBGV. The results of the TREC and TENDS for Photo Realistic terrain portrayal concept shows 
that pilots who where using PR in IMC had a slightly longer recognition time but more time between 
recognition of the problem and the end of their flights than those pilots who used the baseline BSBG in VMC. 
As mentioned earlier, the above results were not strong enough in statistical power and significance. However, 
the trend observed from the above results could indicate that using higher quality terrain portrayal concepts, 
such as EBG and PR in IMC, might have provided evaluation pilots the same or better terrain situation 
awareness and also early warning as the out the window visual cues by the BSBG in VMC. 



TPC 

Figure 36: Mean Recognition Time for each TPC 
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Block Interviews 


Recall that in the first block of runs, combinations of terrain portrayal concepts and guidance symbology 
concepts were presented to each one of the evaluation pilots. During the second block of the runs, the basic 
SVS displays (no guidance, no tunnel) were tested. After the completion of both blocks two batteries of 
questionnaires were administered, each specific to each block. Appendix C shows a copy of both Block 
Questionnaires. Accordingly, the first battery of the questionnaires was specific to guidance, terrain, and 
terrain/guidance interactions. The second battery of questionnaires focused on the terrain portrayal concepts 
and the value of the basic SVS display. Both packages covered a series of workload, situation awareness, and 
preferences related questions. 

Results of Block One Questionnaires 

Here the evaluation pilots were shown all display combinations tested and were asked to answer the 
questionnaires for each display combination. In the first block of questionnaires, pilots were asked to answer 
the questions related to TPC independent of any GSC. In the second block, they were asked about the 
guidance symbology concepts regardless of TPC used. 

Answers Related to Terrain Portrayal Concepts 

Figure 37 shows the level of workload perceived by evaluation pilots for different terrain portrayal concepts 
independent of GSC. Note the ratings for workload were color coded from very low to excessive and the color 
scale in the pie charts starts from light blue to darker blue for low to acceptable and continues towards darker 
red for the excessive rating. Overall, one can observe that the colors of perceived workload for BSBG and 
CCFN are much darker and more towards red than those of EBG and PR are. This means (consistent with 
TLX results) that the perceived workload for PR and EBG was lower than for CCFN and BSBG. 
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Subjective Workload Estimate - Blue Sky-Brown Ground 

□ Very Low □ Slightly High 

□ Low ■Moderately High 

□ Moderately Low BHigh 

■ Slightly Low ■ Very High 

■ Acceptable ■ Excessive 


Subjective Workload Estimate - Constant Color Fish Net 

□ Very Low □ Slightly High 

□ Low ■ Moderately High 

□ Moderately Low BHigh 

■ Slightly Low ■ Very High 

■ Acceptable ■ Excessive 



37a) Workload Question, BSBG 37b) Workload Question, CCFN 


Subjective Workload Estimate - Elevation Based Generic 


Subjective Workload Estimate - Photo Realistic 

□ Very Low □ Slightly High 


□ Very Low □ Slightly High 

□ Low ■ Moderately High 


□ Low ■ Moderately High 

□ Moderately Low BHigh 


□ Moderately Low BHigh 

□ Slightly Low ■ Very High 


■ Slightly Low ■ Very High 

■ Acceptable ■ Excessive 


■ Acceptable ■ Excessive 



37c) Workload Question, EBG 


37d) Workload Question, PR 


Figure 37: Levels of Overall Workload Perceived by Each TPC 
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TPC Most Preferred 

□ Blue Sky - Brown Ground 

□ Constant-Color Fish-Net 

□ Elevation Based Generic 
■ Photo Realistic 



TPC Least Preferred 

□ Blue Sky - Brown Ground 

□ Constant-Color Fish-Net 

□ Elevation Based Generic 
■ Photo Realistic 


5 . 56 % 



38a) Most Preferred TPC 38b) Least Preferred TPC 

Figure 38: Overall Preferences for TPC 

Next, Figure 38 shows the overall EP preferences for each TPC regardless of GSC. This summary chart for 
terrain portrayal concepts indicates the dislike for BSBG and CCFN, and clearly shows the pilots’ preferences 
for EBG and PR terrain portrayal concepts. One interesting result was the exact split of preference between 
EBG and PR as the most preferred TPC. In fact, this exact split between EBG and PR was maintained within 
pilot groups only for the VFR pilots, while most of IFR pilots preferred PR over EBG and H-IFR preferred 
EBG over PR. 


Answers Related Guidance Symbology Concepts 

Figure 39 shows the level of workload perceived by evaluation pilots for different guidance symbology 
concepts independent on the TPC. Again, the ratings for workload start from very low to excessive. Overall, 
one can observe that the perceived workload for NGNT concept and PRFD were much higher (red/darker) than 
those of other concepts. The lowest workload ratings were observed for CBT. In fact, looking at Figure 40, it 
can be seen that a clear relationship exists between the workload ratings and preferences. Here again the CBT 
shows the highest EP preference rating. As it was seen from the results of the between-run measures (both 
subjective and objective), the guidance and symbology concepts with path based guidance cue and a tunnel 
showed the highest SA and lowest FTE and workload measures. Within this preferred subgroup of guidance 
and symbology concepts, CBT showed even better block scores than CFTGP. Recall that in addition to the 
differences in the design of the two guidance and symbology concepts (CBT versus CFTGP) the size of the 
tunnel for CBT matched exactly the vertical and lateral boundaries of the CDIs while the size the CFTGP 
tunnel was almost twice the CDIs. While the tunnel was not intended to indicate desired or required 
performance boundaries, EP’s tendency was to associate the CDI limits (max deflections) with the lateral and 
vertical dimensions of the tunnel. This partially accounts for some of the SA results as well. One other reason 
cited by evaluation pilots (see also the summary of EP comments in the next section) was the difficulty by H- 
IFR pilots with the Crows Feet Tunnel. 
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Subjective Workload Estimate - NGNT 

□ Very Low □ Slightly High 

□ Low U Moderately High 

□ Moderately Low BHigh 

■ Slightly Low ■ Very High 

■ Acceptable ■ Excessive 


Subjective Workload Estimate - PRFD 

□ Very Low □ Slightly High 

□ Low ■ Moderately High 

□ Moderately Low ■High 

■ Slightly Low ■ Very High 

■ Acceptable ■Excessive 


5.56% 



5.56% 



39a) Workload Question, NGNT 


39b) Workload Question, PRFD 


Subjective Workload Estimate - CFT 

□ Very Low □Slightly High 

□ Low ■Moderately High 

□ Moderately Low ■High 

■ Slightly Low BVery High 

■ Acceptable ■Excessive 


5.56% 5.56% 



Subjective Workload Estimate - UBT 

□ Very Low □ Slightly High 

□ Low ■ Moderately High 

□ Moderately Low ■High 

■ Slightly Low ■ Very High 

■ Acceptable ■ Excessive 


5.56% 



39c) Workload Question, CFT 


39d) Workload Question, UBT 
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Subjective Workload Estimate - GP 

□Very Low □ Slightly High 

□ Low ■ Moderately High 

□ Moderately Low ■High 

■ Slightly Low ■Very High 

■Acceptable ■ Excessive 



Subjective Workload Estimate - C5T 

□ Very Low □ Slightly High 

□ Low ■ Moderately High 

□ Moderately Low ■High 

■ Slightly Low ■ Very High 

□ Acceptable ■ Excessive 



39e) Workload Question, GP 


39f) Workload Question, CBT 


Subjective Workload Estimate - CFTGP 

□ Very Low □ Slightly High 

□ Low ■ Moderately High 

□ Moderately Low ■High 

■ Slightly Low ■ Very High 

■ Acceptable ■ Excessive 


5 . 56 % 



39g) Workload Question, CFTGP 

Figure 39: Levels of Overall Workload Perceived by Each GSC 
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GSC Most Preferred 


GSC Least Preferred 

■ PRFD 


■ PRFD 

□ GP 


□ GP 

□ UBT 


□ UBT 

■ CFT 


■ CFT 

□ CBT 


□ CBT 

■ CFTGP 


■ CFTGP 

■ NGNT 


■ NGNT 


5 . 56 % 



5 . 56 % 



40a) Most Preferred GSC 40b) Least Preferred GSC 

Figure 40: Overall Preferences for GSC 


Results of Block Two Questionnaires 

Evaluation pilots were shown all four basic SVS displays without any guidance and any tunnel and were asked 
to answer the questionnaires for each terrain portrayal concept. In the first set of questionnaires pilots were 
asked to rate their overall impression of the TPC (Figure 40). Here, BSBG in VMC and CCFN in IMC were 
clearly dominated by inadequate rating and EBG and PR in IMC were perceived as more towards adequate. 
This might seem rather surprising that the BSBG in VMC was rated lower than PR and EBG in IMC. Pilots’ 
strong preference for EBG and PR might have overshadowed the relevance of OTW in their rating of their 
overall impression of terrain portrayal concepts. Also recall that the OTW presented to the pilots was based on 
the PR terrain projected on the GAWS front wall. The only difference between the PR displayed on the 
primary flight display and the one in OTW was the Minification Factor. 
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Overall Impression - BSBG (VMC) 

■ Totally Inadequate 

■ Very Inadequate 

□ Barely Inadequate 

□ Borderline 

□ Barely Adequate 

□ Very Adequate 

■ Totally Adequate 



5 . 56 % 


41a) BSBG in VMC 


Overall Impression - CCFN 

■Totally Inadequate 
■Very Inadequate 

□ Barely Inadequate 

□ Borderline 

□ Barely Adequate 
■Very Adequate 
■Totally Adequate 



41b) CCFN in IMC 


Overall Impression - EBG 


Overall Impression - PR 

■Totally Inadequate 


■ Totally Inadequate 

■Very Inadequate 


□ Very Inadequate 

□ Barely Inadequate 


□ Barely Inadequate 

□ Borderline 


□ Borderline 

□ Barely Adequate 


□ Barely Adequate 

■Very Adequate 


□ Very Adequate 

■Totally Adequate 


■ Totally Adequate 



5 . 56 % 



41c) EBG in IMC 


41c) PR in IMC 

Figure 41: Overall Impression of TPC, NGNT 


In the second set, evaluation pilots were asked to rate the information provided by different terrain portrayal 
concepts for lateral distance from terrain (Figure 42). Here again, BSBG in VMC and CCFN in IMC were 
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rated as less adequate than EBG and PR in IMC. This is consistent with the FTE results shown in Figure 29 
for lateral path deviation. 


Lateral Distance - BSBG (VMC) 


Lateral Distance - CCFN 

■Totally Inadequate 


■ Totally Inadequate 

■Very Inadequate 


□ Very Inadequate 

□ Barely Inadequate 


□ Barely Inadequate 

□ Borderline 


□ Borderline 

□ Barely Adequate 


□ Barely Adequate 

■Very Adequate 


■ Very Adequate 

■Totally Adequate 


■ Totally Adequate 




42a) Lateral Distance, BSBG in VMC 


42b) Lateral Distance, CCFN in IMC 


Lateral Distance - EBG 

■ Totally Inadequate 

□ Very Inadequate 

□ Barely Inadequate 

□ Borderline 

□ Barely Adequate 

■ Very Adequate 

■ Totally Adequate 


5 . 56 % 



42c) Lateral Distance, EBG in IMC 

Figure 42: Lateral Di 


Lateral Distance - PR 

■ Totally Inadequate 

□ Very Inadequate 

□ Barely Inadequate 

□ Borderline 

□ Barely Adequate 

□ Very Adequate 

■ Totally Adequate 



42d) Lateral Distance, PR in IMC 

from Terrain, NGNT 
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Similarly, response to questions related to the third set, information regarding the vertical distance from terrain 
(Figure 43), was also consistent with the results of SA (Figure 33). 


Vertical Distance - BSBG (VMC) 


Vertical Distance - CCFN 

■ Totally Inadequate 


■ Totally Inadequate 

□ Very Inadequate 


■ Very Inadequate 

□ Barely Inadequate 


□ Barely Inadequate 

□ Borderline 


□ Borderline 

□ Barely Adequate 


□ Barely Adequate 

■ Very Adequate 


□ Very Adequate 

■ Totally Adequate 


■ Totally Adequate 



43a) Vertical Distance, BSBG in VMC 43b) Vertical Distance, CCFN in IMC 


Vertical Distance - EBG 

■ Totally Inadequate 

□ Very Inadequate 

□ Barely Inadequate 

□ Borderline 

□ Barely Adequate 

□ Very Adequate 

■ Totally Adequate 



Vertical Distance - PR 

■ Totally Inadequate 

□ Very Inadequate 

□ Barely Inadequate 

□ Borderline 

□ Barely Adequate 

□ Very Adequate 

■ Totally Adequate 

5 . 56 % 



43c) Vertical Distance, EBG in IMC 43d) Vertical Distance, PR in IMC 

Figure 43: Vertical Distance from Terrain, NGNT 
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The fourth set of questions was related to the information provided by terrain portrayal concepts regarding the 
forward range from the terrain (Figure 44). It seems that the evaluation pilots could gain more information 
related to forward range with CCFN than with BSBG in VMC but the highest scores were again given to EBG 
and PR. 


Forward Range - BSBG (VMC) 


Forward Range - CCFN 

■ Totally Inadequate 


■ Totally Inadequate 

■ Very Inadequate 


■Very Inadequate 

□ Barely Inadequate 


■ Barely Inadequate 

□ Borderline 


□ Borderline 

□ Barely Adequate 


□ Barely Adequate 

□ Very Adequate 


■ Very Adequate 

■Totally Adequate 


■Totally Adequate 



5 . 56 % 


44a) Forward Range, BSBG in VMC 44b) Forward Range, CCFN in IMC 


Forward Range - EBG 

■ Totally Inadequate 

□ Very Inadequate 

□ Barely Inadequate 

□ Borderline 

□ Barely Adequate 

□ Very Adequate 

■ Totally Adequate 


5 . 56 % 



Forward Range - PR 

■ Totally Inadequate 

■ Very Inadequate 

□ Barely Inadequate 

□ Borderline 

□ Barely Adequate 

□ Very Adequate 

■ Totally Adequate 



44c) Forward Range, EBG in IMC 


44d) Forward Range, PR in IMC 


Figure 44: Forward Range from Terrain, NGNT 
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The next sets of questionnaires dealt with workload and SA as shown in Figures 45 and 46, respectively. The 
perceived SA and workload scores were consistent with the results of SART shown in Figure 33 and TLX in 
Figure 34, respectively. Again, PR and EBG scored better than CCFN for both SA and workload. 


Workload - BSBG (VMC) 

■ Totally Unacceptable 

■ Very Unacceptable 

□ Barely Unacceptable 

□ Borderline 

□ Barely Acceptable 

□ Very Acceptable 

■ Totally Acceptable 



Workload -CCFN 

■ Totally Unacceptable 

□ Very Unacceptable 

□ Barely Unacceptable 

□ Borderline 

□ Barely Acceptable 

□ Very Acceptable 

■ Totally Acceptable 


5 . 56 % 



45a) Workload Question, BSBG in IMC 


45b) Workload Question, CCFN in IMC 


Workload - EBG 

■ Totally Unacceptable 

■ Very Unacceptable 

■ Barely Unacceptable 

□ Borderline 

□ Barely Acceptable 

□ Very Acceptable 

■ Totally Acceptable 



Workload - PR 

■ Totally Unacceptable 

■ Very Unacceptable 

□ Barely Unacceptable 

□ Borderline 

□ Barely Acceptable 

■ Very Acceptable 

■ Totally Acceptable 



45c) Workload Question, EBG in IMC 45d) Workload Question, PR in IMC 

Figure 45: Perceived Workload, NGNT 
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Situation Awareness - BSBG (VMC) 


Situation Awareness - CCFN 

■ Totally Inadequate 


■ Totally Inadequate 

■ Very Inadequate 


□ Very Inadequate 

□ Barely Inadequate 


□ Barely Inadequate 

□ Borderline 


□ Borderline 

□ Barely Adequate 


□ Barely Adequate 

□ Very Adequate 


□ Very Adequate 

■ Totally Adequate 


■ Totally Adequate 



5 . 56 % 


46a) SA Question, BSBG in VMC 


46b) SA Question, CCFN in IMC 


Situation Awareness - EBG 


Situation Awareness - PR 

■ Totally Inadequate 


■ Totally Inadequate 

□ Very Inadequate 


HVery Inadequate 

□ Barely Inadequate 


□ Barely Inadequate 

□ Borderline 


□ Borderline 

□ Barely Adequate 


□ Barely Adequate 

□ Very Adequate 


□ Very Adequate 

■ Totally Adequate 


■ Totally Adequate 




46c) SA Question, EBG in IMC 


46d) SA Question, PR in IMC 


Figure 46: Perceived Situation Awareness, NGNT 


The next set of questions was specific to the how well the TPC provided the information about the extreme 
topography (Figure 47). Here again both EBG and PR were rated higher than CCFN and even BSBG in VMC. 
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Extreme Topography - BSBG (VMC) 


Extreme Topography - CCFN 

■ Totally Unacceptable 


■ Totally Unacceptable 

□ Very Unacceptable 


□ Very Unacceptable 

□ Barely Unacceptable 


□ Barely Unacceptable 

□ Borderline 


□ Borderline 

□ Barely Acceptable 


□ Barely Acceptable 

□Very Acceptable 


□ Very Acceptable 

■ Totally Acceptable 


■ Totally Acceptable 




47a) Topography Question, BSBG in VMC 


47b) Topography Question, CCFN in IMC 


Extreme Topography - EBG 

■ Totally Unacceptable 

□ Very Unacceptable 

□ Barely Unacceptable 

□ Borderline 

□ Barely Acceptable 

□ Very Acceptable 

■ Totally Acceptable 


Extreme Topography - PR 

■ Totally Unacceptable 

■ Very Unacceptable 

□ Barely Unacceptable 

□ Borderline 

□ Barely Acceptable 

□ Very Acceptable 

■ Totally Acceptable 


5 . 56 % 



47c) Topography Question, EBG in IMC 47d) Topography Question, PR in IMC 

Figure 47: Recognition of Extreme Topography, NGNT 


Finally, evaluation pilots were asked if they would like to see any added features to the four terrain portrayal 
concepts, Figure 48. The suggested options for the added features were: a) no additions needed, b) it needs 
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tunnel guidance, c) it needs guidance cue, and d) it needs both tunnel and guidance cue. A careful observation 
of the pie charts shows that pilots liked to see both tunnel and a guidance cue for all terrain portrayal concepts. 
Surprisingly, CCFN seemed to be unacceptable without any guidance and tunnel (no green area). 


Add Features - BSBG (VMC) 


Add Features - CCFN 

□ Nothing 


□ Nothing 

□ Tunnel Only 


■Tunnel Only 

□ Guidance Only 


■ Guidance Only 

■ Both Guidance & Tunnel 


■ Both Guidance &Tunnel 



5 . 56 % 


48a) Added Features Desired, BSBG in VMC 48b) Added Features Desired, CCFN in IMC 


Add Features - EBG 


Add Features - PR 

□ Nothing 


□ Nothing 

■Tunnel Only 


■Tunnel Only 

■ Guidance Only 


■ Guidance Only 

■ Both Guidance &Tunnel 


■ Both Guidance &Tunnel 



48c) Added Features Desired, EBG in IMC 48d) Added Features Desired, PR in IMC 

Figure 48: If Any Added Features Needed, NGNT 
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Exit Interviews - Results of Evaluation Pilot-Preferences Questionnaires 

At the conclusion of the experiment and structured questionnaires, evaluation pilots were shown the pictures of 
the 28 possible combinations of terrain portrayal concepts and guidance and symbology concepts and were 
asked to rank their most favorite display first as number one, second most favorite as number 2, and etc. A 
summary of the average results and standard deviation of this subjective measure is plotted in Figure 49 and 
the numerical results are shown in table 2, 
below. 



□ Mean 
■ StD 


Figure 49: Overall EP Ranking, Order of Preferred Display Combination from 1-28 
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Legend 

GSC 


Guidance Only 
Tunnel Only 
Guidance &Tunnel 
NGNT 


Rank 

Display Concept 

Mean 

StD 

1 

EBG & CBT 

2.33 

1.99 

2 

PR & CBT 

2.93 

2.71 

3 

EBG & UBT 

4.53 

3.44 

4 

PR&UBT 

5.20 

3.59 

5 

CCFN & CBT 

6.47 

3.02 

i 6 

EBG & CFTGP 

9.13 

5.22 

7 

PR&GP 

9.33 

4.20 

8 

PR & CFTGP 

9.60 

4.39 

9 

CCFN & UBT 

9.73 

4.56 

10 

EBG & GP 

9.87 

5.19 

11 

BSBG & CBT 

12.47 

5.15 

12 

CCFN & GP 

12.73 

3.90 

13 

CCFN & CFTGP 

13.00 

4.24 

14 

BSBG & UBT 

13.93 

4.99 

15 

EBG & PRFD 

15.20 

6.00 

16 

PR & PRFD 

15.27 

5.48 

17 

BSBG & GP 

16.33 

5.47 

18 

BSBG & CFTGP 

16.80 

4.84 

19 

CCFN & PRFD 

17.40 

4.73 

20 

PR & CFT 

18.00 

4.63 

21 

EBG & CFT 

18.07 

4.88 

22 

CCFN & CFT 

20.00 

4.11 

23 

BSBG & PRFD 

21.33 

3.96 

24 

BSBG & CFT 

22.60 

2.38 

25 

EBG & NGNT 

24.67 

2.16 

26 

PR & NGNT 

24.73 

1.83 

27 

CCFN & NGNT 

26.60 

1.30 

28 

BSBG & NGNT 

27.73 

1.03 


Table 2: EP Overall Ranking, Order of Preferred Display Combinations from 1-28 

Best ranking scores were registered for displays with both guidance cue and tunnel in combinations of PR and 
EBG. Next, displays with tunnel were favored over displays with guidance only in conjunction with PR and 
EBG. Note that all displays without guidance cue and tunnel ranked the lowest. 
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Summary Results Based on a Glass Cockpit Classification 

In the following an attempt is made to establish a measurable comparison between the Blue Sky Brown 
Ground, as the existing glass cockpit technology (baseline BSBG), and SVS concepts, tested here, as the future 
technology. Thus, the results of the above subjective and objective measures were reorganized based on five 
possible TPC/GSC combinations and are discussed in this section. The first category selected was the 
combination of Pitch/Roll Flight Director and BSBG as a representative of current glass cockpit displays, i.e. a 
typical PFD with flight director (PRFD-BSBG). The second category was based on the PRFD with the three 
SVS terrain portrayal concepts used in this experiment (PRFD-SVS). This category represents a class of new 
SVS displays with a traditional guidance concept (in development by some avionics manufacturers). The third 
category was comprised of all guidance and symbology concepts except PRFD in combination with BSBG 
(HITS-BSBG). This category represents a class of glass cockpits that display some kind of pathway or HITS 
on a typical PFD without terrain display. The fourth category was based on all guidance and symbology 
concepts (utilized in this experiment) except PRFD in combination with all terrain portrayal concepts except 
BSBG (HITS-SVS). This represents the state of the art for SVS glass cockpits. The fifth/last category was 
based on the results of BSBG with VMC like OTW view. This category should represent recently popular GA 
glass cockpits that use PFD with no guidance cue during VFR operations. Recall that the OTW display (in this 
experiment) was based on the Photo Realistic terrain database with a Minification Factor (MF) of unity, while 
the MF for PR display was about 5. 

Results of FTE, SART, and TLX for these new categories were statistically significant and are shown in 
Figures 50-53. Following statistical data were obtained: (F(4,870) =187.7, P<.05) for LPD; (F(4,870) =231, 
P<.05) for VPD; (F(4,870) =23.9, P<.05) for SART; and (F(4,870) =17.7, P<.05), for TLX. 



PRFD-SVS HITS-SVS 

Glass Cockpit Category 


Figure 50: Lateral Path Deviation for Different Display Categories 
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250.007 



PRFD-SVS HITS-SVS 

Glass Cockpit Category 


Figure 51: Vertical Path Deviation for Different Display Categories 



PRFD-SVS HITS-SVS 

Glass Cockpit Category 


Figure 52: Computed SA for Different Display Categories 
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PRFD-SVS HITS-SVS 

Glass Cockpit Category 


Figure 53: Perceived Workload for Different Display Categories 


A close analysis of these figures shows that except for BSBG-VFR, the PRFD-BSBG resulted in highest flight 
technical errors and workload with lowest situation awareness. The addition of terrain to flight director 
(PRFD-SVS) results in some improvement of objective and subjective measures, i.e. FTE and TLX are lower 
and SA is higher for PRFD-SVS as compared to PRFD-BSBG. The next larger improvement in FTE, TLX, 
and SA occurs when the HITS is added to the primary flight display. Next level of improvement in all three 
measures is obtained when SVS terrain is added to the HITS-BSBG (now HITS-SVS). The HITS-SVS 
category resulted in lowest FTE, highest SA, and lowest workload among the five categories flown in IMC. 
These findings reinforce earlier results that SVS displays improve pilot performance while reducing the 
workload and increasing SA as compared to other glass cockpit combinations. Post hoc analysis of SA and 
TLX showed that the SA and TLX for HITS-SVS were in the same group as the results of BSBG-VFR. One 
could conclude that SVS displays used in IMC provide the pilots with the same level of perceived situation 
awareness and workload as a baseline primary flight display used in VMC while drastically improving pilot 
performance, as evidenced by lower flight technical errors. 
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Summary of Evaluation Pilots’ Comments 


During the entire experiment process (during the runs, between the runs and during the exit interviews), all 
evaluation pilot conversations and comments were recorded digitally, have been transcribed, and “cleaned up” 
by the use of correct abbreviations and removal of any participant names. These transcripts may possibly be 
published as a separate document; however, they are readily available to the reader upon request to the authors. 

In addition to the digitally recorded comments, the experiment observers/conductors logged their own notes of 
the pilots’ comments as they understood them at that moment in time. Samples of EP comments, as extracted 
from conductors’ notes during the runs, are shown in tables Fl-14 in Appendix F. 

In the following a summary of evaluation pilots’ comments based on experiment conductors’ notes are 
presented. The summary is subdivided into terrain portrayal concepts, guidance symbology concepts, and 
TPC/GSC interactions related comments: 

Comments related to TPC: 

BSBG was inadequate for SA, difficult in combination with PRFD 

CCFN was better than BSBG but inadequate in many cases, especially CCFN with PRFD or CFT. 

EBG was easy to understand and favored by many, just as good as PR type TPC. 

PR was felt to be the most natural presentation of outside world, some pilots liked EBG better. 


Comments related to GSC: 

PRFD was least favored GSC, especially by VFR pilots. 

CFT corners were difficult to see, but it provided good SA to some pilots. 

GP caused high workload to follow it, but very helpful in reducing FTE. 

UBT provided, overall, good SA, but hard to center, and was lacking a path based cue. 

CFTGP led to very good FTE results and favored by many evaluation pilots. 

CBT was the most favorite guidance as the tunnel continued to the destination, the size of the tunnel 
matched the course deviation indicators and the guidance box was easy to follow. 

Comments about Combination of TPC and GSC: 

Only a few pilots were concerned about combinations such as CFT on CCFN and very few saw any 
interactions between the TPC and GSC. This also supports the results of subjective and objective measures. 
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Conclusions 


A fixed based simulation of four Synthetic Vision System terrain portrayal concepts and seven guidance 
symbology concepts on a primary flight display was conducted and employed three groups of evaluation 
pilots. The scenario was a challenging mountain pass maneuver reflecting an advanced application of SVS 
technology in instrument meteorological conditions. A total of 28 combinations of the above independent 
variables were flown (randomly) twice by each of the 18 evaluation pilots. 

The four terrain portrayal concepts evaluated included the baseline PFD with blue sky/ brown ground 
depiction, Constant Color Fishnet, Elevation Based Generic, and Photo Realistic terrain portrayal. The seven 
guidance symbology concepts tested included Crows Feet Tunnel, Unconnected Boxes Tunnel, connected 
boxes tunnel, Ghost Plane, Pitch/Roll Flight Director, Crows Feet Tunnel with Ghost Plane and no guidance 
no tunnel condition. The effect of guidance symbology concept and terrain portrayal concept was statistically 
significant for both lateral and vertical path deviations, calculated situation awareness rating, and computed 
NASA Task Load Index. The Elevation Based Generic terrain produced the statistically lowest vertical path 
deviation, indicating an increased awareness of flight path and terrain resulting from this terrain portrayal 
concept. While this result was statistically significant with an approximate 10% reduction in error, it was 
potentially not operationally meaningful (3 ft). Photo Realistic and Elevation Based Generic terrain portrayal 
concepts produced the lowest lateral path deviations with an approximate 14% reduction in error. However, 
while the reduction in lateral path error may also not be operationally significant, it does indicate a positive 
response to the terrain stimulus. These results may be attributed to the effective integration of terrain 
information, as intuitively provided by the Elevation Based Generic and Photo Realistic concepts with the 
various guidance concepts. Furthermore, the Elevation Based Generic concept may provide important terrain 
information to the pilot in an easier to assimilate format than other terrain portrayal concepts, leading to 
improved vertical path deviation results. These results are different from previous experiments with SVS 
terrain, which had indicated only an improvement in pilot situation awareness, and are likely due to the 
extreme presence of terrain with the evaluation task (i.e. maintain 500 ft AGL in a mountain pass). 

The lowest flight path deviation was achieved by the use of Crows Feet Tunnel with Ghost Plane, Connected 
Boxes Tunnel, and Ghost Plane (pathway based) guidance symbology concepts. These guidance symbology 
concepts reduced flight technical error by approximately 50% compared to the Pitch/Roll Flight Director, 
demonstrating the enhanced pilot control provided. The Unconnected Box Tunnel, improved lateral path 
deviation over the Pitch/Roll Flight Director, but was not significantly different from the Pitch/roll Flight 
Director for vertical path deviation. The Crows Feet Tunnel performed statistically worse than the Pitch/Roll 
Flight Director for vertical path deviation. This could be based on the very fragmented comer elements 
(crows’ feet) and the lacking of a focused information bundle. The promise of a tunnel and Flight Path Marker 
combination presentation is that the pilot, in a glance, can observe own ship position, orientation, and 
trajectory trend relative to the current and future path. In contrast to a flight director where the awareness has 
to be built-up and inferred through observation of the course deviation indicators, flight director symbology, 
attitude indicator, and probably heading and tum-rate over a period of time. Unlike PRFD, the information 
was readily available by crows feet concept but not perceived easily or quickly as other tunnel concepts. 

Qualitative measures of pilot situation awareness and workload followed the quantitative results. The Crows 
Feet Tunnel with Ghost Plane, Unconnected Boxes Tunnel, and Connected Boxes Tunnel all provided 
significantly superior situation awareness to the other guidance symbology concepts tested. This reflects effect 
of the path preview provided by these tunnel concepts. The connected box and Unconnected Box Tunnel 
concepts produced lower workload results with an approximate 30% reduction compared to the Pitch/Roll 
Flight Director and Crows Feet Tunnel concepts. The Crows Feet Tunnel with Ghost Plane and Ghost Plane 
guidance symbology concepts also significantly reduced workload from the Pitch/Roll Flight Director and 
Crows Feet Tunnel concepts, but to a lesser extent. It should be noted that the Crows Feet Tunnel with Ghost 
Plane and Ghost Plane guidance symbology concepts also improved vertical and lateral path deviation. As a 
result, the increased workload due to the Crows Feet Tunnel with Ghost Plane and Ghost Plane guidance 
symbology concepts did indicate some associated benefit in terms of improved pilot performance. A major 
impact of the terrain portrayal concepts was on pilot situation awareness. For the Elevation Based Generic and 
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Photo Realistic terrain portrayal concepts, higher situation awareness and reduced workload was observed for 
all pilots as compared to blue-sky/brown ground baseline as well as the Constant Color Fishnet terrain 
portrayal concepts. The level of SA provided by constant-color fishnet was statistically higher than BSBG but 
less than EBG and PR terrain portrayal concepts. 

There were no statistically significant interactions found between terrain portrayal or guidance symbology 
concepts for any of the objective and subjective variables considered. This suggests that the SVS terrain may 
have similar benefits for all guidance symbology concepts tested. 

In general, all pilot groups performed similarly well when using Crows Feet Tunnel with Ghost Plane, 
Unconnected Boxes Tunnel, and Connected Boxes Tunnel and Ghost Plane (with vertical path deviation less 
than 30 ft and lateral path deviation less than 50 ft). With the same SVS training provided to all three groups, 
low time VFR pilots performed as well as IFR pilots in the low altitude en-route scenario (in IMC) evaluated 
for this study. 

All SVS displays achieved increased pilot situation awareness, lower flight technical errors and workload as 
compared to the baseline Pitch/Roll Flight Director. Also, the same level of situation awareness and workload, 
with drastic reduction in flight technical error, was achieved for all SVS displays in IMC as compared to the 
blue-sky/brown ground (baseline) display in VMC. These results and results of the pilot preferences clearly 
demonstrate the potential and the effectiveness of SVS displays to enable advanced IFR operations by IFR 
rated pilots. 
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APPENDIX A: Flight Director Control Logics 


The block diagrams of the Pitch/Roll Flight Director’s control logics, as implemented in this experiment, are 
provided in Figures A.l and A. 2. Tables A.l and A.2 are the summary lists of the required input and output 
parameters of the algorithms. Note that variables marked “configurable” were adjustable through the configuration 
file and the GUI interface in the SYS software. 


PATH COMMAND FLIGHT DIRECTOR 020208 

A1 Ragsdale Unisys Corp w . a. ra gsdale @ 1 arc .n as a . gov x46438 


Track Track 
0.033 cmd error 



ROLL 

FLIGHT 

DIRECTOR 


Figure A.l: Roll Flight Director Block Diagram 
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0.017 


Gamma 

cmd 


Gamma 

error 


0.3 Nz cmd 



DIRECTOR 


Figure A.2: Pitch Flight Director Block Diagram 



Table A.l: Roll Flight Director Parameters 


Symbol 

Type 

Description 

Elite 

Variable(s) 

Required 

Configur- 

able* 

Unit/ 

Value 

Lateralerror 

Input 

Ownship lateral distance 
closest to the flightpath 

fLatitude, 

fLongitude 

No 

ft 

Path track 

Input 

Heading of flightpath 

NA 

No 

deg 

Track angle 

Input 

Actual ownship track angle 

fTrueTrack 

No 

deg 

Tuming_roll 

Input 

Desired roll angle (for curved 
path only) 

NA 

No 

deg 

V total 

Var 

True airspeed 

fTAS 

No 

ft/sec 

G 

Const 

Gravitational acceleration 

NA 

No 

32.2 

ft/sec 2 

Tumradius 

Var 

Turning radius from the 
flightpath 

NA 

No 

ft 

Rollangle 

Input 

Ownship actual roll angle 

fRoll 

No 

deg 

Offset gain 

Const 

Lateral error gain 

NA 

Yes 

0.033 

deg/ft 

Track gain 

Const 

Track error gain 

NA 

Yes 

2.0 

X 

Const 

Roll flight director filter time 
constant 

NA 

Yes 

0.25 sec 

* Configurable variab 

es are highlighted in yellow. 
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Table A.2: Pitch Flight Director Parameters 


Symbol 

Type 

Description 

Elite 

Variable(s) 

Required 

Configurable* 

Unit/ 

Value 

vertical_error 

Input 

Current Ownship vertical 
flight path error 

fAltMSL 

No 

ft 

pathgamma 

Input 

Commanded flight path pitch 
angle 

fPitch 

No 

deg 

Gamma 

Input 

Actual flight path pitch angle 

NA 

No 

deg 


Input 

Ownship roll angle 

fRoll 

No 

rad 

e 

Input 

Ownship pitch angle 

fPitch 

No 

rad 

Nzcoord 

Input 

Vertical acceleration of a co- 
ordinated turn = cos . cos 

NA 

No 

g 

Nz 

Input 

Actual vertical acceleration 

Gload 

No 

g 

Alt gain 

Const 

vertical_error gain 

NA 

Yes 

0.017 

deg/ft 

gamma gain 

Const 

Vertical flight path angle gain 

NA 

Yes 

0.6 g/deg 

X 

Const 

Pitch flight director filter time 
constant 

NA 

Yes 

sec 

* Configurable variab 

es are highlighted in yellow. 
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APPENDIX B: Evaluation Pilot Training Syllabus 


Each evaluation pilot was given about 2.1 hours of training. Before starting the training in the simulator a 
ground school type Power Point presentation was conducted explaining each step of the training. Figure B.l 
shows the outline of this presentation. Figure B.2 shows the training objectives and Figure B.3 shows a 
sample of the actual syllabus for a typical scenario. 


Outline 

□ Objectives 

□ Familiarization with General Aviation Work Station (GAWS) Setup: 

• Hardware Components 

• Software Components 


Training Syllabus: 

• Lesson 1 - Baseline BSBG as PFD, 

Air Work 

• 

Lesson 2 - SVS with CCFN as PFD, 

Air Work 

• 

Lesson 3 - SVS with EBG as PFD, 

Air Work 

• 

Lesson 4 - SVS with PR as PFD, 

Air Work 

• 

Lesson 5 - SVS with Pitch/Roll FD, 

Enroute 

• 

Lesson 6 - SVS with Ghost Plane, 

Enroute 

• 

Lesson 7 - SVS with Crows Feet Tunnel, 

Enroute 

• 

Lesson 8 - SVS with, Green Boxes Tunnel, 

Enroute 

• 

Lesson 9 - SVS with Crows Feet and Ghost Plane, 

Enroute 

• 

Lesson 10 - SVS with White Tunnel and Guidance Circle, 

Enroute 


2 

SD-HDD Simulation Experiment B 2008/4/4 


Figure B.l: Outline of EP Training Syllabus 
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Training Objectives 


□ 

Familiarize evaluation pilots with the test set-up. 


a 

Familiarize evaluation pilots with display concepts. 


□ 

Reduce/eliminate learning effects on pilot performance. 


a 

Reduce/eliminate fatigue effects. 


□ 

Use all available display information to minimize pilot flight technical 
errors. 


□ 

Evaluation pilots should fly the simulation just like real flying avoid any 
hazardous terrain or flight situations when they occur. 


□ 

They are expected to communicate their intentions and take corrective 
action if necessary. 

A 

SD-HDD Simulation Experiment B 

4 


Figure B.2: Training Objectives Presented to the Evaluation Pilots 
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Flight Lesson 10: SVS with Crows Feet Tunnel and 

Ghost Plane 


Dual - Local 

Merrill Pass, Alaska 

Recommended Sequence: 

1 . Preflight Orientation 

2. Flight 

3. Post Flight Evaluation 

Lesson Objectives: 

□ Become familiar with a new Guidance 
Symbology Concept (GSC) 

□ Practice visual attitude flight with emphasis on 
precise aircraft control supported by GSC 

Review: 

□ Cockpit management 

□ Use of ND and PFD 

Introduce: 

□ 


Maneuver: 

Fly in the center of the pass maintaining 500 
ft AGL. Refer to the ND and use present 
guidance symbology and the path based 
Course Deviation Indicators (CDI) on your 
PFD. 


• Maneuvers with reference to TPC, GSC and the path 
based CDIs will be conducted for 6 minutes. 

• During this flight the student will gain basic knowledge 
of the GSC for an approach to the field while using TPC to 
enhance Situation Awareness. 

• It is desirable that the student maintain altitude +/-100 
feet, heading +/-10 degrees, bank angle of +/- 10 degrees, 
airspeed +/-10 knots and +/- 1 dot within the CDI 
boundaries. 


Use of Crows Feet Tunnel. 


Post Flight Discussion and 
Training Session Completion 


SD-HDD Simulation Experiment B 


18 

2008/4/4 


Completion Standards: 


Figure B.3: A sample Syllabus for a Typical Training Scenario 
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APPENDIX C: Segmentation Methodology 


The planed flight path had 12 waypoints as shown in Figure C.l. The flight path was broken into three 
segments with a distance-to-go measured from a fixed scenario end. The “end point” and the segment 
distance-to-go measurements are shown in Figure C.2 below. 



Figure C.l: Flight Path Way Points 
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Range: -.4706 


wpl2 


Range: 0 
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Range: 1.253 ^Vp9 
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Range: 1 882 yr Wp7 

Range: 2.587 

' Wp6 


Range: 3.604 

/ Wp5 

Range: 4.144 / W p4 


7 


Range: 5.220 Wp3 


Range: 6.818 


Range: 8.306 


Wp2 


All Ranges in NM 


Merrill Pass 


Wpl 


Figure C.2: Distance-to-go measure 


In the segmented treatment of the runs for FTE, segment 1 included WP 1, WP 2, and WP 3. Segment 2 
contained WP 3, WP 4, WP 5, and WP 6 and finally segment 3 includes WP 6, WP 7, and WP 8. 
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APPENDIX D: Block and Preferences Questionnaires 


Questions related to Block Two runs were posed first. These questions related to the runs with no guidance/no 
tunnel displays, see Figures D.l-8: 


No Tunnel I No Guidance Questionnaire 

1 . T aking into consideration the out-the-window view, the Navigation Display, and the Primary Flight Display, provide your overall 
impression of each display concepts representation of the outside environment when the out-the-window visibility is limited. Please 
explain your responses. 






Blue sky - Brown ground (BSBG) with VMC 


T otally 

Very 

Barely 


Barely 

Very 

T otally 

Inadequate 

Inadequate 

Inadequate 

Borderline 

Adequate 

Adequate 

Adequate 


; J 

Constant Color - Fish Net (CCFN) 


T otally 

Very 

Barely 


Barely 

Very 

T otally 

Inadequate 

Inadequate 

Inadequate 

Borderline 

Adequate 

Adequate 

Adequate 


} 




Elevation Based Generic (EBG) 



T otally 

Very 

Barely Barely 

Inadequate Borderline Adequate 

Very 

T otally 

Inadequate 

Inadequate 

Adequate 

Adequate 


) 




Photo Realistic (PR) 



T otally 

Very 

Barely Barely 

Inadequate Borderline Adequate 

Very 

T otally 

Inadequate 

Inadequate 

Adequate 

Adequate 


) 


END 

1 

BACK 


j NEXT PAGE 1 


Figure D.l: Block Two, NGNT, Overall Impression for TPC 
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L aferai Distance 

2. Taking into consideration the out-the-window view, the Navigation Display, and the Primary Flight Display, provide an estimate of 
how effective each display concept was in helping you to maintain an acceptable lateral distance from the terrain. Please explain 
your responses. 

Blue sky - Brown ground (BSBG) with VMC 






T otally 
Inadequate 

Very 

Inadequate 

Barely 

Inadequate 

Borderline 

Barely 

Adequate 

Very 

Adequate 

T otally 
Adequate 

1 

Constant Color - Fish Net (CCFN) 

■ 

1 

T otally 

Very 

Barely 


Barely 

Very 

T otally 

Inadequate 

Inadequate 

Inadequate 

Borderline 

Adequate 

Adequate 

Adequate 

1 

■ 

Elevation Based Generic (EBG) 

■ 

1 

T otally 

Very 

Barely 


Barely 

Very 

T otally 

Inadequate 

Inadequate 

Inadequate 

Borderline 

Adequate 

Adequate 

Adequate 

1 

■ 

Photo Realistic (PR) 

■ 

1 

T otally 

Very 

Barely 


Barely 

Very 

T otally 

Inadequate 

Inadequate 

Inadequate 

Borderline 

Adequate 

Adequate 

Adequate 


END 

i 

BACK 

i 

NEXT PAGE | 


Figure D.2: Block Two, NGNT, Effectiveness for Maintaining Lateral Distance 
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Vertical Distance 

3. Taking into consideration the out-the-window view, the Navigation Display, and the Primary Flight Display, provide an estimate of 
how effective each display concept was in helping you to maintain an acceptable vertical distance from the terrain. Please explain 
your responses. 

Blue sky - Brown ground (BSBG) with VMC 






T otally 
Inadequate 

Very 

Inadequate 

Barely 

Inadequate 

Borderline Adequate 

| 

Very 

Adequate 

T otally 
Adequate 

1 

1 

Constant Color 

) 

- Fish Net (CCFN) 

1 

1 

T otally 
Inadequate 

Very 

Inadequate 

Barely 

Inadequate 

Borderline Adequate 

Very 

Adequate 

T otally 
Adequate 

1 

1 

; ; 

Elevation Based Generic (EBG) 

1 

1 

T otally 
Inadequate 

Very 

Inadequate 

Barely 

Inadequate 

Borderline Adequate 

Very 

Adequate 

T otally 
Adequate 

1 

■ 

Photo Realistic (PR) 

■ 

1 

T otally 
Inadequate 

Very 

Inadequate 

Barely 

Inadequate 

Borderline Adequate 

Very 

Adequate 

T otally 
Adequate 


END 


BACK 


NEXT PAGE | 


Figure D.3: Block Two, NGNT, Effectiveness for Maintaining Vertical Distance 
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Forward F? an qf 

4. T aking into consideration the out-the-window view, the Navigation Display, and the Primary Flight Display, provide an estimate of 
how effective each display concept was in helping you to judge your forward range to pertinent terrain features. Please explain your 
responses. 






Blue sky - Brown ground (BSBG) with VMC 


T otally 

Very 

Barely 


Barely 

Very 

T otally 

Inadequate 

Inadequate 

Inadequate 

Borderline 

Adequate 

Adequate 

Adequate 


i 


Constant Color - Fish Net (CCFN) 


T otally 

Very 

Barely 

n 

Barely 

Very 

T otally 

Inadequate 

Inadequate 

Inadequate 

Borderline 

Adequate 

Adequate 

Adequate 


i 




Elevation Based Generic (EBG) 



T otally 

Very 

Barely Barely 

Inadequate Borderline Adequate 

Very 

T otally 

Inadequate 

Inadequate 

Adequate 

Adequate 


i 




Photo Realistic (PR) 



T otally 

Very 

Barely Barely 

Inadequate Borderline Adequate 

Very 

T otally 

Inadequate 

Inadequate 

Adequate 

Adequate 


i 


END 

1 

BACK 


NEXT PAGE | 


Figure D.4: Block Two, NGNT, Judging the Forward Range 
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Workload 

5. Taking into consideration the out-the- window view, the Navigation Display, and the Primary Flight Display, provide a 
comprehensive estimate of your workload for each display concept while flying Merrill Pass without guidance or a tunnel. Please 
explain your responses. 

Blue sky - Brown ground (BSBG) with VMC 






T otally 

Very 

Barely 

Borderline Acceptable 

I 

Very 

T otally 

Unacceptable 

Unacceptable 

Unacceptable 

Acceptable 

Acceptable 

i 

Constant Color 

- Fish Net (CCFN) 

■ 

1 

T otally 

Very 

Barely 

Borderline Acceptable 

Very 

T otally 

Unacceptable 

Unacceptable 

Unacceptable 

Acceptable 

Acceptable 

i 

■ 

Elevation Based Generic (EBG) 

■ 

1 

T otally 

Very 

Barely 

Borderline Acceptable 

Very 

T otally 

Unacceptable 

Unacceptable 

Unacceptable 

Acceptable 

Acceptable 

1 

1 

Photo Realistic (PR) 

1 

1 

T otally 

Very Very 

Barely 

d j i- Barely 

Borderline Acceptable 

Very 

T otally 

Unacceptable 

Unacceptable 

Unacceptable 

Acceptable 

Acceptable 


END 


BACK 

1 

NEXT PAGE | 


Figure D.5: Block Two, NGNT, Perceived Workload 
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Situation Awareness 

G. Taking into consideration the out-the-window view, the Navigation Display, and the Primary Flight Display, provide a 
comprehensive estimate of your situation awareness associated with each display concept while flying Merrill Pass without 
guidance or a tunnel. Please explain your responses. 






Blue sky - Brown ground (BSBG) with VMC 


T otally 

Very 

Barely 


Barely 

Very 

T otally 

Inadequate 

Inadequate 

Inadequate 

Borderline 

Adequate 

Adequate 

Adequate 


; J 

Constant Color - Fish Net (CCFN) 


T otally 

Very 

Barely 


Barely 

Very 

T otally 

Inadequate 

Inadequate 

Inadequate 

Borderline 

Adequate 

Adequate 

Adequate 


) 




Elevation Based Generic (EBG) 



T otally 

Very 

Barely Barely 

Inadequate Borderline Adequate 

Very 

T otally 

Inadequate 

Inadequate 

Adequate 

Adequate 


} 




Photo Realistic (PR) 



T otally 

Very 

Barely _ Barely 

Inadequate Borderline Adequate 

Very 

T otally 

Inadequate 

Inadequate 

Adequate 

Adequate 


} 


END 

1 

BACK 


NEXT PAGE | 


Figure D.6: Block Two, NGNT, Perceived Situation Awareness 
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Extreme Tepeerephy 

7. Taking into consideration the out-the-window view, the Navigation Display, and the Primary Flight Display, provide an estimate of how 
acceptable each display concept was in portraying the extreme topography (low valleys and high peaks) of Merrill Pass. Please explain your 
responses. 

Blue sky - Brown ground (BSBG) with VMC 






T otally 

Unacceptable 

Very 

Unacceptable 

Barely 

Unacceptable 

Borderline Acceptable 

| 

Very 

Acceptable 

T otally 
Acceptable 

i 

Constant Color 

- Fish Net (CCFN) 

1 

i 

T otally 

Unacceptable 

Very 

Unacceptable 

Barely 

Unacceptable 

Borderline Acceptable 

Very 

Acceptable 

T otally 
Acceptable 

i 

■ 

Elevation Based Generic (EBG) 

■ 

i 

T otally 

Unacceptable 

Very 

Unacceptable 

Barely 

Unacceptable 

Borderline Acceptable 

Very 

Acceptable 

T otally 
Acceptable 

1 

■ 

Photo Realistic (PR) 

■ 

1 

T otally 

Unacceptable 

Very Very 
Unacceptable 

Barely 

Unacceptable 

D A I' Barel ^ 

Borderline Acceptab | e 

Very 

Acceptable 

T otally 
Acceptable 


END 


BACK 


NEXT PAGE | 


Figure D.7: Block Two, NGNT, Portrayal of Extreme Topology 
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Next, following questions were posed to the EP that relate to Block Two runs, discussing all runs with 
guidance and tunnel, see Figures D.9-17: 
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Tunnel/Guidance Symbology Merrill Pass 

You have flown with the following guidance symbology: Pitch/Roll Flight Director (P/RFD), Ghost A/C with no tunnel 
(Ghost A/C), Green BoxTunnel/no guidance (Box Tunnel), Crows Feet Tunnel/no guidance (Crows Feet Tunnel), 
Connected Boxes Tunnel (Connect Box), Crows Feet tunnel with Ghost A^C guidance (Crows Feet/Ghost) and No 
Tunnel/No Guidance (NoTnoG). 


1 . Which guidance symbology did yo yj^osf prefer 

for the Merrill Pass? 

2. Which guidance symbology did yoi least prefer 

for the Merrill Pass? 


P/RFD 

Ghost 

A/C 


Box 
T unnel 

Crows Feet 
T unnel 

Connect 
T unnel 

Crows 

Feet/Ghost 

NoTnoG 








P/RFD 

Ghost 

A/C 


Box 
T unnel 

Crows Feet 
T unnel 

Connect 

Tunnel 

Crows 

Feet/Ghost 

NoTnoG 


+ 


+ 




3. Explain the reason for the difference. 

4. Provide an estimate of your workload while flying eecf? of the display conditions: 

Pitch/Roll Flight Director (P/RFD) 

Ghost A/C with no T unnel (Ghost A/C) 

Green BoxT unnel (Box T unnel) 

Crows Feet T unnel/no guidance 

I (Crows Feet Tunnel) 

Connected Boxes Tunnel 
(Connect T unnell 

Crows Feet T unnel with Ghost A/C 
guidance (Crows Feet/Ghost) 

No T unnel/No Guidance (NoT noG) 

Very Low Acceptable 

5. Did the use of the funnel concepts significantly enhance situation awarenenss on 
these runs versus the ne tunnef concepts? 



Excessive 


YES 


NO 


END 


BACK 

1 

( NEXT PAGE ] 


Figure D.9: Block One, Tunnel/Guidance, Preferences for GSC 
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Situation Awareness Measure 

1. Indicate the display concepts providing the most situation awareness in your PFD when flying the Merrill Pass. 
When making your choice consider only the display concept against the display concept it is paired with. 


Absolute Very Strong Strong 


Weak 


P/RFD 

P/RFD 

P/RFD 

P/RFD 

P/RFD 

P/RFD 

Ghost A/C 
Ghost A/C 
Ghost A/C 
Ghost A/C 
Ghost A/C 


Equal 

ii 


Weak 


Strong Very Strong Absolute 


Ghost 

A/C 

Boh 
T unnel 

Crows Feet 
T unnel 





Crows 
Feet/Ghost 

NoTnoG 


Boh 
T unnel 

Crows Feet 
T unnel 



Crows 
Feet/Ghost 

NoTnoG 


END 


BACK 


NEXT PAGE 


Situation Awareness Measure 

Continued. Indicate the display concepts providing the most situation awareness in your PFD when flying the Merrill Pass. 
When making your choice consider only the display concept against the display concept it is paired with. 


Absolute Very Strong Strong 


Weak 


Boh Tunnel 

1 ■ 

■ 

' 

' 

' 

■ 

' 

' 

Boh Tunnel 

■ ■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

Boh Tunnel 

1 ■ 

■ 

' 

' 

' 

■ 

' 

' 

Boh Tunnel 

, 

■ 

■ 

■ 

■ 

■ 

■ 

■ 


T unnel 
Crows Feet 

■ ■ 








T unnel 
Crows Feet 

■ ■ 








T unnel 

'■ ■ 








Connect 









T unnel 









CnnnRnl 









Crows Feet/ 

' 1 








Ghost 

1 ■ 









Equal 

J~ 

J- 

J- 

J- 


Weak 


Strong Very Strong Absolute 




Crows Feet 
T unnel 



Crows Feet/ 
Ghost 

NoTnoG 



Crows Feet/ 
Ghost 

NoTnoG 


Crows Feet/ 
Ghost 


NoTnoG 


NoTnoG 


END 


BACK 


NEXT PAGE 
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Figure D.10: Block One, Tunnel/Guidance, Situation Awareness for All Combinations 


Terrain Portrayal ( M em iiPass) 

You have flown with the following terrain portrayals: Blue Sky / Brown Ground (BSBG), Constant Color with 
Fishnet (CCFN), Elevation-based Generic (EBG), Photo Realistic (PR) 


1 . Which terrain portrayal did you jsww/ /me/st 

for the Merrill Pass? 

2. Which terrain portrayal did you Awf /we/ef? 

for the Merrill Pass? 

3. Explain the reason for the difference. 

4. Provide an estimate of your workload while flying 
Blue Sky Brown Ground (BSBG) 

Constant Color with Fish Net (CCFN) 
Elevation Base Generic (EBG) 

Photo Realistic (PR) 




Very Low 


Acceptable 


Excessive 


END 


1 

BACK 


NEXT PAGE 



Figure D.ll: Block One, Tunnel/Guidance, Preferences for TPC 
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Situation Awareness Measure 


Continued: Indicate the display concepts providing the most situation awareness in your PFD when flying the Merrill Pass. 
When making your choice consider only the display concept against the display concept it is paired with. 

Absolute Very Strong Strong Weak Equal Weak 

Strong 

Very Strong Absolute 

H 


CTFN 


i 

■ ■ ■ i 

FR R 


i i . 

■ ■ ■ i 

PR 

BSBG J 

■ 

■ ■ ■ i 

rrFM 


frr 


i 

■ I I 1 

pr 

rrFM j 

EBG 


■ I I 1 

PR 

, , , , , , , , , rc , , , , , , , , 

END 

( BACK ] 

NEXT PAGE 



Figure D.12: Block One, Tunnel/Guidance, Situation Awareness for TPC 
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1 . Rank your preference of the guidance symbology 
you would most like to have in your PFD during ■: 

marginal VFR or IFR weather conditions. 



2 

. Make your choices based on the auidance svmboloav only 
1 regardless of terrain concept. 


5 




|| ^ J - ~t 

Drag from left to right. If an image is lost click 
RESET. 

3 


6 


Hi *>■ 

L* 

4 

RESET 


7 




“ r 

. 

BACK 


[ NEXT RAGE 


Figure D.13: Block One, Tunnel/Guidance, Preferences for GSC 


85 






1 . Rank your preference of the terrain concept you 
would most like to have in your PFD during marginal 
VFR or IFR weather conditions. 


2 

Make your choices based on the terrain concept only 
regardless of guidance symbology. 


Drag from left to right. If an image is lost click 
RESET. 


RESET | 



Exit I 


BACK 

1 

NEXT PAGE | 


Figure D.14: Block One, Tunnel/Guidance, Preferences for TPC 
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3. Indicate the display concepts from choice 1 to choice 20 based on which concept had the fiesf /ea/uves. 
Use numbers 1 through 28 to indicate your order of preference with 1 = most preferred and 28 = least preferred. 


□ 





l_ r PR Ghost r PR Box RR Crows PR c onnect p R Crows PR 

Tunnel l Feet Tunnel 1 Tunnel 1 F^Ghn*r 1 Nr, 


Feet/Ghost NoTnoG 


EGG EGG 

F EBG P/FlFD fl Ghost |Ht BGB | 0X F Crows Feet gFlconnect |^EBG Crows p EBG 
Wm 1 A/C ^Tunnel b Tunnel ^Tunnel ^ Feet/G host ^ NoTnoG 


r- CCFN r- CCFN Ghost CCFN Box CCFN Cmm CCFN 

rp/RFD E..C r w, CStSBj ™' 1 KISrr-H 


Feet/Ghost NoTnoG 


BSBG 


BSBG 


P/RFD 1 Ghost A/C 


r BSBG Box BSBG BSBG 

1 Tunnel T Crows Feet p Connect 
Tunnel B Tunne | 


I — BSBG Crows , — BSBG 
Feet/Ghost NoTnoG 


T ake into consideration situation awareness and 


Select Dn 


workload in your rankings and please explain your 
choices. 



END 

RESET 

j 

BACK 

NEXT PAGE | 


Figure D.15: Block One, Tunnel/Guidance, Ranking for All Combinations 
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5. Indicate your preferences from choice 1 to choice 28 based on which of the display concepts that would he/p the mesf in avoiding a Controlled 
Flight into T errain (CFIT) event. Use numbers 1 through 28 to indicate your order of preference with 1 = most preferred and 28 = least preferred. 


□ 





r pr p/rfd r 


PR Ghost 
A/C 


PR Boh 
T unnel 


|— PR Crows |— PR Connect |— PR Crows |— PR 

Feet Tunnel Tunnel Feet/Ghost NoTnoG 


r ebg p/rfd r 


EBG Ghost 
A/C 


_ _ _ n EBG EBG 

|r t BG B r r Crows Feet "connect 
Tunnel Tunnel Tunnel 


I — EBG Crows ■ — EBG 
Feet/Ghost NoTnoG 


CCFN 

P/RFD 


r- CCFN Ghost r— CCI-N box |XFN Q om _ rrcK . r rrcK . 

' A/C F Tunnel T T _, t C °"" ect CCFN Crows ■“ CCFN 


CCFN Boh 
T unnel 


Feet Tunnel 


CCFN 
Conner 
T unnel 


I— L-L-rN urows ■— 

Feet/Ghost NoTnoG 


w- BSBG 
1 P/RFD 


r BSBG r BSBG I 

Ghost A/C Tunnel 

Take into c 


BSBG Boh bsbg 
T unnel T Crows Feet p 
T unnel 

n situation awareness 



BSBG 
Connect 
T unnel 


|— BSBG Crows |— BSBG 
Feet/Ghost NoTnoG 


Select Dn 



enplain your choices. 
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Figure D.16: Block One, Tunnel/Guidance, Ranking for All Combinations for CFIT Avoidance 
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6. Indicate your preferences from choice 1 to choice 20 based on which of the display concepts that would help the mesf in avoiding a Low Visibility, 
Loss of Control (LVLoC) event. Use numbers 1 through 28 to indicate your order of preference with 1 = most preferred and 28 = least preferred. 
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Figure D.17: Block One, Tunnel/Guidance, Ranking for All Combinations for LVLOC Avoidance 


Finally, if you have any comments that will help us in evaluating the display concepts, 
improving the experiment or issues concerning the GAWS implementation please provide 
them. 


Thank you for providing all this important information in bringing 
the best technologies to the general aviation community! 
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Figure D.18: Final Question 
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APPENDIX E: Unusual Runs and Outliers 


A total of 48 runs, (6 GSC) X (4 TPC) X (2 Replicates) = 48, were conducted in the block one group, i.e. runs 
using the display combinations with GSC. There were 4 actual CFIT incidents and 8 runs where the 
evaluation pilots encountered temporary loss of SA without loss of control or CFIT. Table E.l shows the 
details of these unusual runs, including comments recorded by the experiment Principal Investigator (PI). The 
table indicates that, among terrain portrayal concepts tested here, there were no unusual runs for the EBG. 
However, there were 6 unusual runs for BSBG, 4 for CCFN, and 3 for PR. Among the guidance and 
symbology concepts, one can see that evaluation pilots had problems with PRFD (5 unusual runs), 5 for CFT, 

3 for GP, and none for UBT, CBT or CFTGP. As it will be shown in the next sections UBT, CBT and CFTGP 
were also the most favored guidance and symbology concepts by the pilots. Consistently, Elevation Based 
Generic was most favored terrain portrayal concept by evaluation pilots. 


TPC 

GSC 

Rating 

Comments by EPs from PI Notes. PI Comments in Italic. Runs with GSC 

Source 

BSBG 

CFT 

VFR 

Can't tell where the boxes are. Lost this one. It's difficulty flying wings level. 

SA 

BSBG 

PRFD 

VFR 

Had a hard time. Was looking at the FD and tried to stay with it and than lost the CDI and the 
other way. Would not be able to fly this one well. Quite frustrating and stressful. PI= EP 
almost lost it before the turn but recovered. 

SA 

BSBG 

PRFD 

IFR 

VMC, I looked outside more and lost the track of PRFD. 

SA 

BSBG 

PRFD 

VFR 

(I was) Using the ND to try to fly the path. Chased one needle and lost the other. SA was all 
OTW and ND. Zoomed in ND to fly the path closer thought that is not supposed to be the 
guidance. Pretty difficult situation even OTW, the bowl of valley especially. 

SA 

BSBG 

CFT 

IFR 

Went very high for 15 sec and then came back. Difficult to stay in the center. Lost the tunnel 
reference could be due to the on set of fatique. Needed to stay on top all the time and it was 
difficult to make a turn. 

SA 

BSBG 

PRFD 

IFR 

Chasing needles all over the place. Very sensitive. Guidance lost and got real closed to the 
wall. Most difficult by far. Seemed to be very sensitive and difficult to anticipate. Had a hard 
time following it. 

SA 

CCFN 

CFT 

IFR 

No bad but seems would have been easier with the ghost plane. Got lost in the turn because of 
the colors of the tunnel. 

SA 

CCFN 

GP 

IFR 

PI=He didn't pay attention to the terrain and crashed in .6 minutes, CFIT 

CFIT 

CCFN 

GP 

HIFR 

FOV=30 is useless for CFT. PI=CFIT in the middle of the turn. 

CFIT 

CCFN 

CFT 

VFR 

got to side and lost the tunnel. Tried to get back had CFIT at WP3, 1 can see the symbology 
but I just couldn't get cought up to it. 

CFIT 

PR 

PRFD 

HIFR 

Struggling to stay and was correcting in the wrong direction. CFIT right before the turn. 
(Turned around 180 then crashed. Lost elevator control.) 

CFIT 

PR 

CFT 

IFR 

Still hard today on the CFT. Lost in the comer and had some spatial disorientation. It is okay 
in a straight line. It was the turn that was difficult. 

SA+SD 

PR 

GP 

IFR 

Lost the ghost a couple of times. 

SA 


Table E.l: Loss of SA and Controlled Flight into Terrain Cases for Runs with GSC 


Unusual No Guidance No Tunnel Runs 

Table E.2 shows the list of the unusual runs among the 144 NGNT runs (4 terrain portrayal concepts * 2 
Replicates * 18 evaluation pilots). Here, there were 4 CFIT runs with CCFN and 2 with EBG. Among the 
unusual NGNT runs, there were not any cases with PR (in IMC) or BSBG in VMC. 
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TPC 

GSC 

Rating 

Comments by EPs from PI Notes. PI Comments in Italic. No Guidance No Tunnel Runs 

Source 

CCFN 

NGNT 

VFR 

Not able to determine my distance above ground caused the CFIT. PI= CFIT in 3:51 minutes. 

CFIT 

CCFN 

NGNT 

IFR 

PI= too low. Stopped 2.4 into flight. 

CFIT 

CCFN 

NGNT 

IFR 

Finding the pass with this concept is difficult. SA was low due to lack of depth cue on the 
CCFN. PI= CFIT in 3:40 minutes into the flight. 

CFIT 

CCFN 

NGNT 

HIFR 

PI= CFIT before WP4. 

CFIT 

EBG 

NGNT 

IFR 

Looking outside doesn’t help much. PI= CFIT in 3:40 minutes 

CFIT 

EBG 

NGNT 

HIFR 

PI= CFIT during the turn. 

CFIT 


Table E.2: Controlled Flight into Terrain Cases for Runs with No Guidance and No Tunnel 


Segmented Treatment 

During the segmented treatment of FTE there were few unusually large values of FTE for some of these 
segments. A careful examination of these values showed that the second tier unusual data were based on a 
single run. In this particular run, the EP lost his lateral SA as he was “distracted” by the ND. For the analysis 
of segmented treatment of FTE, any data with an outlier larger than 10 times the standard deviation of the 
measures were treated as an outlier and removed from the data bank. Figures E.l - E.12 show the data 
distribution for lateral and vertical path deviations before and after the outlier removal process. 
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400 - 


300- 



Mean =57.3441 
Std. Dev. =49.37965 
N =064 


Figure E.l: Distribution of Lateral Deviation Values for segment 1 before outlier removal 



Figure E.2: Distribution of Lateral Deviation Values for segment 1 after outlier removal 
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400 - 


300- 



Mean =47.3763 
Std. Dev. =61 .53576 
N =864 


Figure E.3: Distribution of Lateral Deviation Values for segment 2 before outlier removal 



Figure E.4: Distribution of Lateral Deviation Values for segment 2 after outlier removal 


93 




400 - 


300- 



Mean =75.91 43 
Std. Dev. =71 .561 13 
N =864 


Figure E.5: Distribution of Lateral Deviation Values for segment 3 before outlier removal 



s3laerms, ft 


Figure E.6: Distribution of Lateral Deviation Values for segment 3 after outlier removal 


94 




400 - 



Mean =27.431 7 
Std. Dev. =28.51 71 6 
N =864 


Figure E.7: Distribution of Vertical Deviation Values for segment 1 before outlier removal 



Figure E.8: Distribution of Vertical Deviation Values for segment 1 after outlier removal 
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400 - 


300- 



Mean =30.1 311 
Std. Dev. =27.501 29 
N =864 


Figure E.9: Distribution of Vertical Deviation Values for segment 2 before outlier removal 



Figure E.10: Distribution of Vertical Deviation Values for segment 2 after outlier removal 
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400 - 



Mean =26.9055 
Std. Dev. =25.36573 
N =864 


Figure E.ll: Distribution of Vertical Deviation Values for segment 3 before outlier removal 



Figure E.12: Distribution of Vertical Deviation Values for segment 3 after outlier removal 
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APPENDIX F: Samples of Evaluation Pilot Comments 


Samples of EP comments, as extracted from PI notes during the runs, are shown in tables Fl-14. The first 
group of tables shows the comments of H-IFR evaluation pilots, followed by comments of IFR and finally 
VFR pilots. In each group of tables, the comments are grouped by the blocks of terrain portrayal concepts 
followed by the NGNT runs. 

As mentioned earlier the full transcripts of the audio taped EP comments will be (possibly) published as a 
separate document, however, these are readily available to the reader upon request from the authors. 
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TPC 

GSC 

Comments 

HIFR 



BSBG 

PRFD 

1 tend to over-control quite a bit. No terrain information, that is not helpful. Flard to anticipate the turn 
but looked at the ND an noticed the turn. Don't like these things. Over-controlling could be due to the 
turbulence. No terrain awareness. It takes a lot of concentration to stay centered up. It jumps a lot. 
Turns are not easy on this one. Even a small turn is hard to tell where to focus. It helps to stay on 
target once you are on. 

BSBG 

GP 

Better than the last display. GP tends to jump a lot and may be it is because 1 am trying to anticipate 
too much. The lateral is the worst. Flard to fly smoothly with the GP, 1 am just chasing it. 1 need to do a 
lot of physical work to chase the GP. 1 feel like over-controlling. Advantage of this display is very little 
clutter. GP requires a lot more control movements to center. Flard to be smooth on the control but is 
also fairly accurate. As long as you keep up with the GP, you are doing okay but very busy. 

BSBG 

CBT 

Would have been helpful to have a center marker on the tunnel. Other than that, the guidance is pretty 
good. Turbulence was pushing him around and hard to stay smooth. Could see outside a little for the 
ground and that is it. Easy to over-control this. No TA. Tunnel looks good. The background is less 
cluttered and can see the background very well. Tunnel allows me to see what is coming up ahead and 
that is helpful. Turbulence really gives some physical demand. 1 can see the turn coming up. 

BSBG 

UBT 

No terrain awareness except the outside. Still 1 like the balloon and the green boxes; it is hard to know 
where the center is. The boxes are smaller and makes it easier than the CFT to find the center. Feel 
like over controlling a little bit. No TA. 1 like the balloons. It helps to see where 1 am without the ND. 
Fairly good guidance. Easy to see the center of the tunnel and the balloons also help. For the green 
boxes you have to stay left in the box for right turn and right in the box when turning left. VMC, my 
tendency is to fly VFR instead of flying the boxes. 1 have to stay outside the boxes around the corner. 

BSBG 

CFTGP 

FOV 30 is not helping much. 1 am chasing it too much. T ry to go to the center of the box to reduce over 
controlling and chasing the GP. 1 don't care for BSBG it doesn't tell me much about terrain. 1 seems to 
be high workload with this and no terrain. 

BSBG 

CFTGP 

Little clutter on the background and thus good contrast with the CFT. CFT helps me to see where the 
future path is. Lot of physical control manipulation needed to stay with the it. 

BSBG 

CFT 

CFT is hard to tell where the center is. No TA. Gives good understanding of where the path is but is 
not very accurate. Easy to get off course. 1 am getting off but not realized until it happened. The active 
CFT disappears behind the tapes. Without the GP, the CFT alone makes it harder to fly and hard to 
judge the center. Better than nothing but not too great. EP scanned the CDIs more than GP. 

BSBG 

NGNT 

VMC ; 1 am learning MFD better, the altitude is a wild guess. 


Table F.l: A Sample of EP (H-IFR) Comments for the Combination of BSBG and GSC. 
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TPC GSC Comments 

HIFR 

It is hard to judge 500 ft. I try to stay in touch with the ground. I don't know what altitude it is suppose to 

CCFN NGNT be but I still try to keep close to the ground. 

In the first turn it gets hard to tell how high you are, it seems relatively low. It is the way the CCFN is 
building up, which make you look closer to the ground. It is actually not a bad thing. Once I learned to 

CCFN GP fly the GP, it is little bit easier to anticipate where it is going. 

CFT helped somewhat knowing where the path leads. With the GP, I tend to move the control yoke 
more than the others but it is more accurate. With 30 FOV this gives pretty good TA but not as good as 
CCFN CFTGP the EBG and the PR. It is not bad. 

A lot more demand on my scan work and I need to look at the ND more. TA was moderate. Once 
getting the cross-hair lined up it is okay. The issue is when making the turn, ND is needed. The bars 

CCFN PRFD jump quite a bit. The noodle on the ND makes it look like you need to turn all the time. 

Some terrain awareness. Don't like the CFT by themselves; it is hard to see where the center of the 
boxes is. Hard to see where you are in space. TA is decent. Usable but not very good compared to the 

CCFN CFT others. Hard to tell when you are drifting. 

There is some TA with this. It takes some physical manipulation to keep the GP in the center. It is too 
sensitive. Turbulence makes the thing jump around. Hard to stay with the GP. Really I feel like over- 

CCFN GP controlling the airplane with the GP. 

Hard to see the CFT corners. It would have been helpful to have some kind of marker at the center of 
the Crows Feet Tunnel. There is some TA though not as good as the other TPCs. Hard to see the 

CCFN CFT Crows Feet center also. Kind of confusing on where the velocity vector is going to go. 

Moderate TA. The CFT makes it easy to know where the path is but use the GP to stay in the center. 
CFT helps to see the turn. The terrain display makes me feel I was too low; may be because of the 
CCFN CFTGP lines. TA with this CCFN is not as good as with the others. 


Good TA. The FD seems somewhat easier on this combination but it jumps around and hard to be 
smooth. It is not as physically and as intensive as GP but it is awfully sensitive. Got to cross-reference 

CCFN PRFD with the ND but still it is better than the previous display. 

Probably leaving FOV at 60 when flying all the time. Balloons allow you to see the center of the tunnel. 
It would be nice to have some sort of marker at the center of the tunnel. I tend to stay outside of the 

CCFN UBT boxes during a turn. Relatively easy to fly. 

Moderate terrain information. Not much cue on the terrain and it appears lower than actually it is. 
Magenta box sometimes disappears behind the tapes and makes it more difficult to see th center of 

CCFN CBT the tunnel. Hard to see center of the magenta box when it is obscured by the tapes. 

Moderate terrain awareness. Like the balloon. Staight and leve is easy, turns are more difficult. I like 
the heading information, it is easy to read. I need to stay on the outside of the box during the turns. 

CCFN UBT Tendency is to cut inside to much. 

Turbulence really bounced me around. Hard to tell where is the exact center of the boxes. Good 
awareness of the terrain. Feel like I was over-controlling just to keep up with it. Easy to fixate on the 
CCFN CBT white box ahead and thus not targeting at the right point. 

Table F.2: A Sample of EP (H-IFR) Comments for the Combination of CCFN and GSC. 
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TPC 

GSC 

Comments 

HIFR 



EBG 

GP 

Good TA; especially the 30 FOV is pretty accurate compared to the OTW. 1 like this terrain. It is good. 1 
like the 30 FOV at least in the first turn. It is impossible to see with the 30 FOV on the other displays. 
Tougher to stay with the plane at 60 FOV. 

EBG 

CBT 

1 like this display, FOV=30 is good for terrain clearance. Wind and turbulence makes it harder to follow 
the box, 1 killed the terrain and white boxes. It is hard to see the center of the box, a cross-hair would 
have been nice in the center of the box. 

EBG 

CBT 

1 like this terrain it gives me visual of outside and terrain elevation. There is a tendency to cut through 
the tunnel but the guidance box helps to stay on it. FOV30 terrain is better but 1 need 60 to stay in the 
box. The OTW is not enough 1 am relying on PFD. 

EBG 

CFTGP 

Crows Feet helped me to get back inside the box. 1 am still learning to follow it with the help of ND. 1 go 
to FOV of 30 which 1 like for terrain but 1 tend to over control with 30. 1 try to fly smooth but 1 can't 
because 1 am trying to stay out of terrain. 

EBG 

GP 

Good TA. A lot of physical movements to keep the ghost plane in the center. Flard to be smooth but 
seems to be very accurate. 30 FOV is too sensitive to follow the plane well. Easy to visualize the turn. 
Tend to over-correct after the turn. 

EBG 

UBT 

1 like the balloons; don't have to look at the ND. The terrain awareness is good. 1 like this kind of terrain 
display. So far, when going straight, the system is pretty good. 1 could see the boxes ahead. Some 
tendency to shoot for far away boxes than the one in front of me. 

EBG 

UBT 

Do like the waypoint balloons. Like the green boxes though it is hard to tell when it is the transition to 
the next green box. Tendency is to cut the corners on this one. As you need to visually stay to the 
outside of the boxes as it comes towards you. 

EBG 

CFTGP 

Good terrain. Seems like still have to over control a little but perhaps due to a new day; not too bad. 
This is not a bad concept. The terrain awareness is pretty good. 30 FOV feels like really know where 
the mountains are. 1 see the big turn coming up. 

EBG 

CFT 

1 don't like the CFT as guidance. It is hard to tell where you are. Contrast between the CFT and the 
terrain makes it hard to see (corners) them. They do help to see the turn coming up. Tendency is to 
cut the corners especially during the turns. 

EBG 

PRFD 

Good TA. This 30 FOV is quite comfortable for PRFD. The lateral guidance is a lot more jumpy than 
the vertical guidance. Cross-checking the ND quite a bit. Pretty good TA. With 30 FOV 1 can see how 
close the mountains are. Tendency to pull up. 

EBG 

PRFD 

Lateral FD seems to be overly sensetive. You have to rely on ND for turn information. 1 like this terrain 
presentation. 

EBG 

NGNT 

1 am more comfortable about this terrain still difficult to say hight above the ground. 


Table F.3: A Sample of EP (H-IFR) Comments for the Combination of EBG and GSC. 
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TPC 

GSC 

Comments 

HIFR 



PR 

CFTGP 

Good TA. The airplane is easy to follow once you get to the middle of the CFT. Takes a little effort to 
get used to it. The PR helps to some extend. It gives better orientation awareness. A lot of physical 
effort to keep the GP centered but once centered it works good. 

PR 

PRFD 

1 normally don't like FD but here it is nice. 

PR 

UBT 

Good TA. Easy to stay in the center during the straight path. Like the balloon which helps to stay on 
path. Tendency to cut the turns again. Overall it is a pretty good display; good PR, good control. But 
during the turn it was hard to figure out where it is. 

1 1 

PR 

PRFD 

Good TA. The cross-hairs are pretty sensitive. Turbulence is hard to follow. PR display makes it more 
comfortable, knowing where the terrain is. Cross hairs are jumpy and 1 need to use ND. Needle does 
not help much. Too much turn indications then what is needed. 

PR 

GP 

It is a little bit difficult to pick the altitude from the display so 1 have a hard time to get started right. 1 
have to cross-check with ND to see if the turn is coming up. It is hard to stay with ghost plane. With 
visual cues of PR, 1 can anticipate what they are. 

PR 

CFT 

Good TA because of the PR. Don't like the CFT by themselves very much. It is hard to tell where the 
center of the CFT is, plus the other symbolgys also clutter the center of the tunnel. The ND noodle 
seems to be more than it should be. 

PR 

CBT 

Excellent terrain awareness. The TA is good for 30 FOV but 1 can't see the magenta box thus 
switching back to 60. Good display probably the nicest combination so far at least for the straight and 
level. Still hard to see where the center is. 

PR 

CBT 

Good TA. This is a decent display with the tunnel. Pretty compelling view and give a good security on 
where you are along the flight path. Pretty good set up and still know where the terrain is, even though 
the 60 FOV distorts the reality. The tunnel let you know where you are going. 

PR 

CFT 

Advantage is the magenta contrast well with the PR and you can see the magneta much better than 
the other terrain. It is still not as accurate to follow as the other GSc. It does give you an idea where 
you are going and good TA. 

PR 

GP 

Like the 30 FOV but GP jumps around too much. 60 is not bad. Pretty good SA but need to look at the 
ND to anticipate the turns. Once getting used to the idea that the plane will have to make a violent turn, 
it becomes easier. 

PR 

NGNT 

Good realism; 1 am confident in where 1 am. Green boxes still let me stay to the outside of the turns. 1 
like this display. One needs to get used to it. it seems like the box shows one is in the middle but not 
the CDIs. 

PR 

NGNT 

Good TA. Great for terrain awareness for sure without the guidance. 


Table F.4: A Sample of EP (H-IFR) Comments for the Combination of PR and GSC. 
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TPC 


Comments 

IFR 






BSBG 

CBT 

1 stay with FOV60, 

BSBG 

UBT 

1 used 30 of FOV because boxes were slower. The BSBG made me look out the window more even 
though there was not much to see. Seeing outside gave me a more secure feeling. 1 focused more on 
the second rather than the first box in front of me. 

BSBG 

CFT 

FOV of 60 had more clutter and a lot of disconnected lines all over. The CFT lines lead to less SA 
because it is hard to see where 1 was going and 1 started to go way off. It was better, less clutter, with 
BSBG than other TPCs but more risky. 

BSBG 

CFTGP 

It is not bad with the CFT to assist the GP. With just the CFT, 1 don’t like to zoom-in but with this one 
the zoom-in helps me to keep closer track with the GP and with less clutter. 

BSBG 

CFTGP 

The background requires glancing out the window more. The OTW made me think 1 was closer. 1 
wasn't quite ready for some quick changes in time. 1 tried to look outside more. 1 was lagging behind 
the GP a little and was not aware of what is outside. 

BSBG 

CFT 

Without the background ( terrain ), 1 had to worry more how close to the mountain 1 was. 1 felt like 
chasing the CDI more. 1 wasn't totally sure without glancing the ND to see where 1 was. The CFT not 
connected, especially in the turn, It was hard to see where to turn. 

BSBG 

UBT 

1 try to keep focus on the immediate box in front of me. 1 need to glance over to the ND to see the 
waypoint numbers. The green box is unconnected and thus harder to get an idea on where you are 
going. It is harder to notice the boxes. They don't give much speed change. 

BSBG 

CBT 

With tunnel 1 can see what is happening and allowed me to see and control the speed better. 1 could 
see the tunnel going up. Not having the background it was more disconcerting. 

BSBG 

GP 

Without the background ( terrain ), 1 looked more at the ND and took attention away from the PFD. 
Chase plane required a lot of chasing especially during the turn and right after it. The GP made a lot of 
directional changes at that point and made me work harder. 

BSBG 

GP 

It is a little more difficult to anticipate the GP. The initial setup took a little bit to get on. 1 could see the 
(ghost) plane but still it took a little to get on at the beginning. Not having the background (terrain) 1 
needed to rely on GP. 

BSBG 

PRFD 

With no background (terrain) 1 needed to look at the ND more but was not able to do it because 1 was 
too busy. 1 had a tough time especially at the sharp turn. Too focused on the FD and 1 was not able to 
know what is happening. Very little anticipatted. 

BSBG 

PRFD 

VMC, 1 looked outside more and lost the track of PRFD. 

BSBG 

NGNT 

VMC, 1 was looking at OTW more and used my airspeed for my altitude indicator. 


Table F.5: A Sample of EP (IFR) Comments for the Combination of BSBG and GSC. 
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TPC 


Comments 

IFR 






CCFN 

CBT 

30 FOV doesn't help much. That makes me put velocity vector through the white boxes instead. When 
getting real close to the ground, the CCFN appears closer than it actually is. The guidance part feels 
good to me with 60 FOV. 

CCFN 

CBT 

It was better with the CCFN. It was easier to see. 1 like the boxes and it is easier to keep up with the 
boxes and see where to go. Once 1 was in the box, attention was okay. 1 can see the lines going up 
and down on the terrain. 1 took focus away from ND. 

CCFN 

CFT 

When you get close to terrain they merge and you can't see the contour lines. With the CFT, it is easy 
to go off the CDI and thus it required more work. You loose a little bit of what is going on due to the 
blending of the terrain. 

CCFN 

PRFD 

This is simlar to the GP because it is decluttered a little more. 1 felt close to the ground. Everything was 
alright, contours were okay. 

CCFN 

CFTGP 

CCFN was fine, good background info just more difficult with the depth perception but you can see the 
mountains; that helped. GP is a small target but helped to keep it right on the target. Toggled between 
30 and 60 to keep things straight. 

CCFN 

PRFD 

Without indication on where to go it was a little difficult. 1 am all over with this one. When two hill are 
together it is difficult to distinguish them and is confusing. 1 needed to catch up with the FD. 

CCFN 

GP 

The contour map {CCFN) was fine and getting more used to it. 1 am not 100% sure about depth 
perception. GP was hard to anticipate and did not give me enough bank and speed information. 1 say 
below normal in understanding of the situation. 

CCFN 

CFTGP 

1 looked at the CFT and GP together to help anticipate; little depth perception with this background 
{Terrain ) . 1 used ND to verify but it was hard to tell how close the mountains were. 1 was looking mostly 
at the chase plane {GP) but did not worry too much on the precision. 

CCFN 

CFT 

Sometimes the left side of CFT was visible but the right side wasn't; perhaps due to the clutters with 
the tape and CDI. The background (terrain) was not really a huge factor and 1 had a good idea where 1 
was. 

CCFN 

UBT 

With the contour mapping was fine. 1 felt comfortable with the guidance box and it was nice to know 
where the walls were. Reassuring knowing where things were. Trying to get to the center of the next 
box could get into an unusual attitude. 

CCFN 

NGNT 

With the CCFN it is very hard to get your eyes off of PFD. 


Table F.6: A Sample of EP (IFR) Comments for the Combination of CCFN and GSC. 


TPC 


Comments 

IFR 






EBG 

CFTGP 

30 degrees FOV during the turn seems to be helpful for the turn. With chase plane {GP) you can see 
the small adjustments you need to make; The contour on the terrain helped me anticipate where to go. 

EBG 

CFT 

The CFT corners were dragging me out of the CDI. Though 1 was off just a little but the CDI showed a 
lot. The CFT let him know where the turn was and thus 1 was able to anticipate it. It requires huge 
correction with CFT. 

EBG 

CBT 

Better this time in keeping predictor on the magenta box. The background was good to have. No need 
to use 30 FOV to see the white boxes because 1 was not looking at the white boxes anymore for lining 
up the predictor. 

EBG 

CFTGP 

1 cut the corners to catch up with GP. 1 like FOV of 30 here better because it gets me closer to GP (on 
the straight leg). 

EBG 

CBT 

1 put the noodle on the first waypoint (on the ND) and that works quite well. 1 saw a lot of space around 
me on the ND and made me feel comfortable. Compared this with GP, the GP is more precise and that 
keeps you on your toes; the magenta box is not as hard as others. 

EBG 

GP 

1 was checking more the ND but sometimeswhen 1 was looking at the ND and 1 would lose track of the 
GP quickly. 1 was more aggressive on keeping up {with guidance ) because 1 was not sure what the GP 
will be doing next. It was good with the background. Good depth perception with light shadows. 

EBG 

NGNT 

this time 1 was using the terrain to get to the way point. 

EBG 

NGNT 

1 had a better depth perception. It might be due to FOV 30. 


Table F.7: A Sample of EP (IFR) Comments for the Combination of EBG and GSC. 
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TPC 


Comments 

IFR 






PR 

PRFD 

FD does not allow you to prepare how far off you will be to the next waypoint and set up for it. Thought 
1 followed the FD quite closely but the CDI shows 1 was off. The PR background gives good depth 
perception. The FD was difficult to set up at the beginning. 

PR 

PRFD 

With this GSC it was hard to know how far the cross-hair will move and 1 started chasing it around. 
Good to have the terrain but during the turn, 1 could not see where the cross-hair was going go and 
thus 1 put a lot of attention on the cross-hair and did not get the chance to look at the terrain. 

PR 

UBT 

30 degrees of FOV made more emphasis on the balloons and thus 1 decided to stick with the 60. The 
balloons sometimes appear to be at a different place until one is right in front of me and 1 had to ignore 
it. The PR helped when 1 was making a harder turn. 

PR 

CFTGP 

1 like the 30 FOV because it declutters things a little but that makes the control more jerky. It is easier 
to toggle between 30 and 60 than the other GSCs. Sometimes there were too many CFT ; it was 
confusing. Too many lines sometimes it gets confusing. 

PR 

GP 

It was useful to have the GP. More exact with FOV of 30. But 1 was not sure what is the best strategy 
fto anticipate the turn. PR was better in judging how far/high 1 was. It was easier to see how close 1 was 
the the terrain. 1 spent more time chasing the plane. 

PR 

CFT 

The PR was more relaxing and 1 did not go much far off course and it didn't cause the CDI to deviate 
more because 1 did not apply jerky control efforts instead of a calmer control inputs. The PR helped the 
attentional resource and not making too many sharp control inputs. 

PR 

CBT 

It is good with the PR; i am more comfortable to what is around me and it is more realistic. The other 
terrains look fake. PR feels more accurate and comfortable. Boxes show what is coming up and the 
white boxes give you good constrast and depth. 

PR 

UBT 

The PR was good kept me aware of where everything was especially with the green box. 1 aimed at the 
balloons with the velocity vector and threw me off. Waypoint balloons help to know when the waypoints 
are coming. 

PR 

CFTGP 

With PR terrain you can see the river and the slope of the mountains; it gives better depth perception, 
similar to the corners of the terrain on the OTW. 1 am more confident seeing things outside. 1 was 
chasing the GP and was not watching what was around. 


Table F.8: A Sample of EP (IFR) Comments for the Combination of PR and GSC. 
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TPC GSC Comments 
VFR 



BSBG 

UBT 

With BSBG 1 had no idea where is the terrain. More unnerving. It is less likely to anticipate the change 
and thus less stable. 1 had to concentrate quite a bit on the box and had problem picking out which box 
to point to. 

BSBG 

UBT 

In straight flight demand on attention was low but it was harder to stay in the center during a turn. 1 had 
time to look at ND. Boxes provided good guidance by aiming several boxes ahead. 

BSBG 

CBT 

Tunnel laid out the course in front of you, no guess work. Plenty of time to plan ahead. It lacked 
specific terrain detail. 

BSBG 

PRFD 

PRFD needed a lot of attention as you had to go back and forth to try to center the FD. 1 thought it was 
centered but the CDI showed it was off course. 1 stuck with the FD since there were no terrain. 
Demand on attention was thus high. 

BSBG 

PRFD 

It takes some effort to follow FD especially with no terrain. It is difficult to follow terrain with no terrain 
on PFD. Quite a bit of work. 1 did not have much chance to look at ND. It took a lot of concentration to 
follow FD. 

BSBG 

CFTGP 

It was easy to follow the GP. 1 tried to look OTW as much as possible because there was no terrain on 
ND. There was attention to spare due to the GP. 1 had time to look at the ND. FOV of 30 and 60 were 
both usable for the GP. 

BSBG 

CFT 

1 did have spare attention because of the boxes. During the turn it was more difficult because the 
boxes were more difficult to interprete. Only terrain cue 1 had was from OTW but it was limited. Boxes 
don't have sides and tops made them more difficult to interpret. 

BSBG 

CFT 

Demand of attention was high due to the box which caused my deviations several times. Determining 
the angle of the boxes from the CFT was difficult. It took more thought process to evaluate the CFT. 
You needed to figure out where the CFT ( comers ) were. 

BSBG 

GP 

GP was easy to follow but the terrain was not visible except hte OTW. You needed to look over ND 
and that caused the situation to change suddenly. 1 looked outside to check for terrain. GP kept me on 
track but 1 did not have time to do other things. 

BSBG 

CFTGP 

1 could look around without drifting too far from the ghost plane. Understanding the orientation of the 
plane in roll is easier for BSBG. 

BSBG 

CBT 

BSBG gave better horizon or plane roll orientation. 


Table F.9: A Sample of EP (VFR) Comments for the Combination of BSBG and GSC. 


TPC GSC Comments 

VFR 



CCFN 

CFT 

CCFN gave good impression on the terrain. CFT was hard to interprete during a turn. Flard to predict 
the trajectory because the CFT ( corners ) piled up and it was hard to distinguish the boxes thus there 
was less time for other tasks. CCFN gave no problems for terrain clearance. 

CCFN 

UBT 

The display (PFD) was a little jerky at the beginning. The unconnected boxes tunnel was a lot easier 
than the CFT to tell where you are going. 1 could prepare for the next box and had time to look at the 
CDIs. 

CCFN 

PRFD 

Demand on attention was high especially in sharp turns. It was hard to stay on the lines. CCFN helped 
to get an idea where the terrain was without solely relying on the FD. The FD lines move quickly 
especially when they move away from each other, can cause confusion. 

CCFN 

GP 

CCFN generally gives good TP. TA was high but at some locations it is hard to figure out how high you 
are. GP provided everything 1 needed, combined with the CCFN. 

CCFN 

UBT 

Guidance box was easy to follow. CCFN gave plenty of idea on terrain except in a few locations. Turns 
required more mental demand to judge the center of the boxes. Steep turn made it difficult to judge the 
boxes. 

CCFN 

CBT 

Having guidance box always in front plus CCFN, they made the demand on attention low. Tunnel laid 
out the course ahead, which is good. 1 was not worried about the quality of the CCFN becaue the 
guidance information was good. Magenta box in front helped out a lot. 

CCFN 

CFTGP 

CCFN gave enough terrain information. CFT indicated it was clear of terrain. GP stayed right on the 
course. GP does not require too much thought. 

CCFN 

CFTGP 

When close to the terrain with CCFN is difficult to judge distance to it. 

CCFN 

CFT 

This display combination was rather difficult. 

CCFN 

CBT 

Easy to follow the tunnel and never had any lapse on guidance information. 


Table F.10: A Sample of EP (VFR) Comments for the Combination of CCFN and GSC. 
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TPC 


Comments 

VFR 






EBG 

PRFD 

Similar to the previous one. EBG gave plenty of terrain information. Turns took more effort to keep the 
FD on course. EBG allowed you to see the direction with the PRFD. FD was less forgiving than the 
other guidances. 

EBG 

CFTGP 

Easy to follow the GP. CFT helped understand the tunnel. EBG gave clear terrain. No problem with the 
guidance. Easy to see the CFT with EBG. 1 could look at the ND too. 

EBG 

UBT 

There was not much instability. 1 had time to look at the ND. TP was fine and the box was clear in the 
zone. There wasn't any terrain issues, especially with the boxes. At the turn, if you aim at the center of 
the box early, it will be fine. 

EBG 

GP 

1 had time to look at the ND. 

EBG 

GP 

Ghost plane was easy to follow, EBG was easy to see. Sometimes the plane would make a turn, but 1 
did not have to contcentrate on flying the ghost too much and could watch the CDIs too. Ghost does 
not take too much mental demand at all. 

EBG 

CBT 

It was obvious as to where to go. With EBG 1 always knew where the terrain was. The tunnel was 
extended and showed the trajectory making me understand what to do before it happened. It allowed 
time for other tasks. 30 degrees FOV was hard to use and made Guidance Box unviewable. 

EBG 

CFTGP 

CFTGP gave good guidance and EBG gave good preception on terrain. GP plane required more 
attention, especially for VFR. But the combination overall gave enough capacity for other tasks. EBG is 
a good representation of the terrain. 

EBG 

CBT 

Easy to fly through the tunnel and thus easy to prepare for the maneuvers. 1 had a better perception of 
the tunnel due to the lines between them. The combination was good. 1 like the 30 degrees FOV quite 
a bit. The tunnel seemed to approach slower and easier to fly. 

EBG 

PRFD 

Following the FD required some workload. It was easier to follow the needle since EBG displays terrain 
well and made the turn to anticipate easier. 1 had less time to look at the ND. Terrain was observable 
but it was hard to keep track of it since you needed to concentrate on the FD. 

EBG 

CFT 

It is hard to see the CFT during a turn. With EBG it was clear to see the terrain. CFT was difficult at the 
corner. Glancing between PFD and ND required more time. 

EBG 

UBT 

This was with a tunnel, it was easy except 1 didn't have a turn rate to help me. 


Table F.ll: A Sample of EP (VFR) Comments for the Combination of EBG and GSC. 
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TPC 


Comments 

VFR 






PR 

GP 

Good combination, 1 knew where 1 was. GP gave good direction. 1 wsa able to look often to the ND. 
The GP ables me to anticipate. No confusion. 1 understand the orientation of the aircraft and terrain. 
PR is good for mountain but 1 am not sure how dirt and grass will look like. 

PR 

CFTGP 

Ghost plane made it easy to follow the tunnel, you just need to concentrate on it and boxes became 
just the side symbol. CFT gave only little bit of heads up on what will happen but ghost helped a lot and 
1 had time to look at the ND. 

PR 

GP 

GP was easy to follow and it did not take too much attention from other things. With PR 1 was clear to 
know exactly where 1 was. 1 could look at the ND and see the check points. 30 degrees FOV was good 
for better resolution. 

PR 

CFTGP 

1 had plenty of time to look around and at the ND. If you want to center GP precisely it will take some 
attention. Situation was clear with the PR, and GP. CFT (comers) were a little hard to see. GP 
provided most information needed. 

PR 

PRFD 

When the FD was centered, there were too much symobolgy and they cluttered in the center, including 
the water marker and velocity vector. 1 felt more comfortable not because of the change in terrain but 
more practice. 

PR 

CFT 

Little higher attention was demaned than green boxes tunnel (UBT) but not too bad. PR made a big 
difference helping CFT (it made it better). Most of the time 1 could see CFT (corners) but couple of 
times 1 couldn't see them due the the color of the CFT. PR helped. 

PR 

CBT 

No instability, you can an see tunnel in front easly and can do other things. No problems understanding 
the terrain and seeing the obstructions. 1 like the 60 degrees FOV which let me see the mag box fully. 

PR 

PRFD 

It was easier to follow the FD today; it could be due to practice. It made it a lot easier to stick with the 
FD. You needed to use the ND for guidance. PR helped on the guidance with FD. If you looked away 
from FD too long it could deviate quickly from you. 

PR 

CFT 

Similar to the previous one. It takes some effort to use the CFT. The color can get lost in the 
background. 1 had time to look at other displays. SA of a PR is high. It was hard to figure out where the 
boxes are and hard to see the four corners of the boxes. 


PR 

CBT 

Visual cues form the tunnel and magenta box and PR made it easy to track the course. 

PR 

UBT 

Higher attention was demanded than connected boxes. 1 had no magenta box in front so 1 had to aim 
for the further boxes an needed the CDIs to help stay on course. Green boxes made it hard to judge 
where the aircraft was. 


Table F.12: A Sample of EP (VFR) Comments for the Combination of PR and GSC. 
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APPENDIX G: Results of the Flight Technical Errors for Segmented Treatment 


In each one of Figures G.l -9, mean RMS of lateral and vertical path deviations for segments 1, 2, and 3 have 
been plotted far left, center, and right of the figure, respectively. The letters, si, s2, and s3 refer to segment 1, 
segment 2, and segment 3, respectively. The letter combination of Tae’ refers to lateral path deviation and ‘ve’ 
refers to vertical path deviation. Accordingly, sllaerms is the value of the mean RMS of lateral path deviation 
for segment 1, s21aerms the mean RMS of lateral path deviation for segment2, and s31aerms the mean RMS of 
lateral path deviation for segment3. Similarly, si verms refers to mean RMS of vertical path deviation for 
segments 1 and so on. 


Effect of Terrain Portrayal Concepts on Flight Technical Errors for Segmented Treatment 

During segment 1 and segment 3 where the maneuver can be considered a level maneuver the trends for the 
mean RMS of LPDs were similar to the results obtain for the complete scenario, i.e. BSBG and CCFN show 
higher deviations than the EBG and PR but the results are not (statistically) significant. 

Effect of Guidance Symbology Concepts on Flight Technical Errors for Segmented Treatment 

The values of the mean RMS of the LPDs for each segment (sllaerms, s21aerms, and s31aerms) are plotted in 
Figure G.l, below. The values of mean RMS of the VPDs for each segment (slverms, s2verms, and s3verms) 
are plotted in Figure G.2, below. These results match the results of FTE for the un-segmented (entire scenario) 
treatment. 
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Figure G.l: Mean of Lateral Path Deviations for Segments 1, 2, and 3, by GSC 
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Figure G.2: Mean of Vertical Path Deviations for Segments 1, 2, and 3, by GSC 
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Effect of Evaluation Pilot Qualifications on Flight Technical Errors for Segmented Treatment 

The values of the mean RMS of the LPDs for each segment by EP Rating are plotted in Figure G.3 and the 
values of the mean RMS of the vertical path deviations for each segment are plotted in Figure G.4 below. 
Note that the vertical path deviation values by Rating for segment 3 were not (statistically) significant. 
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Figure G.3: Mean of Lateral Path Deviations for Segments 1, 2, and 3, by Rating 
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Figure G.4: Mean of Vertical Path Deviations for Segments 1, 2, and 3, by Rating 


Interaction Effects of Rating and GSC on Flight Technical Errors for Segmented Treatment 

The values of the mean RMS of vertical and lateral deviations are plotted for each TPC and colored by EP 
Rating, Figure G.5 and G.6. Note that the scale of all of the above figures were expanded to 210 ft since the 
mean RMS of LPD for CFT guidance by H-IFR pilots, during segment 3, exceeded the scale of 100 ft used in 
other plots. 
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Figure G.5: Mean of Lateral Path Deviations for Segments 1, 2, and 3, by GSC and Rating 
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Figure G.6: Mean of Vertical Path Deviations for Segments 1, 2, and 3, by GSC and Rating 
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